Brief:
In cloud computing, we have API available from service providers like Azure/AWS which allows us to create and resource (VM,DB,etc.) at run time and delete when the job is done. It is very helpful to save the cost. Using below procedure you can create and delete the Azure HDInsight Cluster automatically through application (Console application) and terminate the instance when the processing is done. Here we are using non interactive Authentication method for sign in.
Need:
As we know, we can create Azure HDInsight cluster through Azure Portal manually. But if we have a requirement where we need to create HDInsight cluster and after finishing the work we need to delete it (to save the cost). Rather than doing this manually we can create an application which can be schedule in scheduler to run daily and it will do required the operations (Create HDInsight Cluster, Delete Cluster) automatically.
How It Works:
Requirements-
- Need a Microsoft Azure Subscription.
- Install NuGet package given below,
- Install-Package Microsoft.Rest.ClientRuntime.Azure.Authentication -Pre
- Install-Package Microsoft.Azure.Management.ResourceManager -Pre
- Install-Package Microsoft.Azure.Management.HDInsight
To manage the HDInsight cluster (Create and Delete) you can use following class.
public class HdinsightClusterManager
{
// The client for managing HDInsight
public HDInsightManagementClient _hdiManagementClient;
private ClusterCreateParameters Parameters { get; set; }
public HdinsightClusterManager(string SubscriptionId, string TenantId, string ClientId, string clientSecret)
{
Parameters = new ClusterCreateParameters();
// Authenticate and get a token
var authToken = Authenticate(TenantId, ClientId, SubscriptionId, clientSecret);
// Get an HDInsight management client
_hdiManagementClient = new HDInsightManagementClient(authToken);
}
public void InitClusterCreateParameters(int NewClusterNumWorkerNodes, string NewClusterUsername, string NewClusterPassword,
string NewClusterLocation, string ExistingStorageName, string ExistingStorageKey, string ExistingBlobContainer,
string NewClusterType, string rdpPassword, string rdpUsername, string _HeadNodeSize, string _WorkerNodeSize)
{
Parameters = new ClusterCreateParameters();
Parameters.ClusterSizeInNodes = NewClusterNumWorkerNodes;
Parameters.UserName = NewClusterUsername;
Parameters.Password = NewClusterPassword;
Parameters.Location = NewClusterLocation;
Parameters.DefaultStorageAccountName = ExistingStorageName;
Parameters.DefaultStorageAccountKey = ExistingStorageKey;
Parameters.DefaultStorageContainer = ExistingBlobContainer;
Parameters.ClusterType = NewClusterType;
Parameters.RdpPassword = rdpPassword;
Parameters.RdpUsername = rdpUsername;
Parameters.RdpAccessExpiry = DateTime.Now.AddDays(10);
Parameters.HeadNodeSize = _HeadNodeSize;
Parameters.WorkerNodeSize = _WorkerNodeSize;
}
public bool CreateCluster(string existingResourceGroupName, string newClusterName)
{
System.Console.WriteLine("Creating a cluster. The process takes 10 to 20 minutes [" + DateTime.Now.ToShortTimeString() + "] ...");
Parameters.OSType = OSType.Windows;
// Create the cluster
_hdiManagementClient.Clusters.Create(existingResourceGroupName, newClusterName, Parameters);
System.Console.WriteLine("The cluster has been created [" + DateTime.Now.ToShortTimeString() + "].");
return true;
}
public OperationResource DeleteCluster(string existingResourceGroupName, string clusterName)
{
System.Console.WriteLine("Deleting a cluster. The process takes 10 to 20 minutes[" + DateTime.Now.ToShortTimeString() + "] ...");
var result = _hdiManagementClient.Clusters.Delete(existingResourceGroupName, clusterName);
System.Console.WriteLine("The cluster has been deleted [" + DateTime.Now.ToShortTimeString() + "].");
return result;
}
/// <summary>
/// Authenticate to an Azure subscription and retrieve an authentication token
/// </summary>
/// <param name="tenantId">The AAD tenant ID</param>
/// <param name="clientId">The AAD client ID</param>
/// <param name="subscriptionId">The Azure subscription ID</param>
/// <param name="clientSecret">Get this from Application created in Active Directory</param>
/// <returns></returns>
static private TokenCloudCredentials Authenticate(string tenantId, string clientId, string subscriptionId, string clientSecret)
{
var authContext = new AuthenticationContext("https://login.windows.net/" + tenantId);
var credential = new ClientCredential(clientId, clientSecret);
var tokenAuthResult = authContext.AcquireToken("https://management.azure.com/", credential);
return new TokenCloudCredentials(subscriptionId, tokenAuthResult.AccessToken);
}
/// <summary>
/// Authenticate to an Azure subscription and retrieve an authentication token by interactive method
/// </summary>
/// <param name="TenantId">The AAD tenant ID</param>
/// <param name="ClientId">The AAD client ID</param>
/// <param name="SubscriptionId">The Azure subscription ID</param>
/// <returns></returns>
static private TokenCloudCredentials Authenticate(string TenantId, string ClientId, string SubscriptionId)
{
var authContext = new AuthenticationContext("https://login.microsoftonline.com/" + TenantId);
var tokenAuthResult = authContext.AcquireToken("https://management.core.windows.net/",
ClientId,
new Uri("urn:ietf:wg:oauth:2.0:oob"),
PromptBehavior.RefreshSession,
UserIdentifier.AnyUser);
return new TokenCloudCredentials(SubscriptionId, tokenAuthResult.AccessToken);
}
/// <summary>
/// Marks your subscription as one that can use HDInsight, if it has not already been marked as such.
/// </summary>
/// <remarks>This is essentially a one-time action; if you have already done something with HDInsight
/// on your subscription, then this isn't needed at all and will do nothing.</remarks>
/// <param name="authToken">An authentication token for your Azure subscription</param>
static private void EnableHDInsight(TokenCloudCredentials authToken)
{
// Create a client for the Resource manager and set the subscription ID
var resourceManagementClient = new ResourceManagementClient(authToken);
//resourceManagementClient.SubscriptionId = SubscriptionId;
// Register the HDInsight provider
var rpResult = resourceManagementClient.Providers.Register("Microsoft.HDInsight");
}
}
To get the required parameter you can follow the below methods,
- SubscriptionId and TenantId – Log in to Azure Portal, if you click on the Help icon in the upper right and then choose 'Show Diagnostics' you can find the SubscriptionId and tenantid in the diagnostic JSON.
- ClientId and clientSecret – To get the ClientId and clientSecret follow the procedure given here and also give the appropriate permission to Azure Resource Group where we will be creating HDInsight Cluster. After initializing HDInsightManagementClient class object you can use CreateCluster and DeleteCluster Methods to Create and Delete the HDInsight Cluster. You can refer to link for remaining parameters. Creation and Deletion of cluster will take around 20 to 30 minutes.