Category Archives: Integration

vRealize Automation 7.3 REST API Documentation

With a lion’s roar, a new version of vRA went GA on May 16th – complete with dozens of new features, hundreds of bugfixes, and a heaping helping of love and care. If you’d like more info on anything but that last part, please see the release notes here. But one thing that may be overlooked are the significant improvements that have been made in the vRealize Automation 7.3 REST API Documentation.

As of the time of writing, we  have made several samples available (in the form of Postman collections) containing REST API calls for our most common vRA use cases.  These samples are hosted on GitHub  at https://github.com/vmwaresamples/vra-api-samples-for-postman

For more detailed information on these samples, please see this blog post by our very own Sudershan Bhandari on what he was trying to accomplish with this collection and how you can use it to accelerate your use of the vRA APIs. https://blogs.vmware.com/management/2017/05/vrealize-automation-api-samples-for-postman.html

Some examples of the API samples provided include:

  • Create and entitle a composition blueprint
  • Create and entitle a parameterized blueprint (using the all-new component profiles)
  • Export/Import blueprints and other content/components
  • Perform various day 2 operations on catalog resource including reconfigure, Scale-In/Scale-Out and others
  • Manage endpoint configuration
  • Create approval policy and approve or reject an approval request
  • Create reservations of various types
  • Create and manage a tenant, including creating authentication directories for the tenant
  • Manage users and their roles
  • Configure a NSX provisioning setup including endpoint, reservation, network profiles and sample blueprints
  • Create property definitions and retrieve values backed by vRO script actions
  • Create and manage reclamation requests
  • Register event topics and subscribe/delete subscription to event topics
  • API tips on bearer token management, pagination, sorting, filtering

We have also entirely revamped our API documentation reference on VMware{code}, so it now shows the APIs per service, an overview of each service, the API listing and relevant sample code snippets all in a very organized and easily searchable manner. Check that out at  https://code.vmware.com/apis/vrealize-automation

Our API programming guides have also been completely reworked for ease of use and friendlier navigation – to get you started faster and support  you more easily.

So, while there are tons of amazing new capabilities in our new flagship release of vRealize Automation, I hope you won’t overlook the huge investment we’ve made in this vital area. Check it out today!

As always, this post was brought to you by Tropikalia IPA by White Stork Brewing Company. It’s pretty much my go-to while I’m working with our amazing vRA engineers in Sofia.

tropikalia_ipa

Using the new Microsoft Azure Endpoint in vRealize Automation 7.2

After months of planning and development, vRealize Automation 7.2 finally went GA today, and it feels so good! One of the most anticipated and spotlight features of this new release was the Endpoint for Microsoft Azure. I had the privilege of working very closely with the team who delivered this capability, and thought I would take some time to develop a brief POC type guide to help get you started using the new Microsoft Azure Endpoint in vRealize Automation 7.2

This guide will walk you through configuring a brand-new Azure subscription to support a connection from vRealize Automation, then help you set up your vRA portal and finally design and deploy a simple Blueprint. We will assume that you have already set up your Azure subscription. If not, you can get a free trial at https://azure.microsoft.com/en-us/free/ – and that you have a vRealize Automation 7.2 install all ready to go. Certain steps outlined in this guide make assumptions that your vRA configuration is rather basic and is not in production. Please use them at your own risk and consider any changes you make before you make them!

Part 1: Configuring Azure

SelectSubscription
Once you have your subscription created, log in to the Azure portal and click on the Key (Subscriptions) icon  in the left-hand toolbar. These icons can be re-ordered, so keep in mind that yours may be in a different spot than mine. Note down the Subscription ID (boxed in red above) – you will need this later!

OpenDiagnosticsTenantID
Next, click on the Help icon near the upper right corner and select Show Diagnostics. This will bring up some raw data about your subscription – and here is the easiest place I’ve found to locate your Tenant ID. Simply search for “tenant” and select the field shown above. Note this ID for later as well.

Now you’ll need to create a few objects in the Azure portal to consume from vRA. One of the great capabilities the new endpoint brings is the ability to create new, on demand objects per request – but to make things a little cleaner we will create just a few ahead of time. We’ll start with a Storage Account and a Resource Group.

CreateStorageAccountAndResourceGroup
Locate the Storage Accounts icon in the sidebar – again, keeping in mind that these icons can be reordered and you may have to poke around a bit to find it. Make sure the correct Subscription is selected and click Add.

You’ll be prompted with a sliding panel (Azure does love sliding panels) where you can fill in some important details about your Storage Account. This is basically a location where your files, VHDs, tables, databases, etc will be stored. Enter a Name for the Storage Account – you’ll need to make sure to follow the rules here. Only lowercase letters, must be globally unique, etc. You can choose to change any of the options presented here, but for the purposes of this guide we will leave the defaults and move on to the Resource Group. This is a logical grouping for deployed workloads and their related devices/items – and to keep things clean, we will specify a new one now. Note the name of this Resource Group for later. You’ll also need to choose a Location for the workloads – pick whatever is convenient or geographically reasonable for you. I chose West US – make a note of this as well! Click Create.

CreateVirtualNetwork
Now, let’s create a simple Virtual Network. Locate the Virtual Network icon on the panel to the left and click it. Ensure the correct Subscription is selected and click Add.

Again, you’ll be prompted with some basic configuration. Enter a unique name for your new Virtual Network and record it for later. You can choose to modify the other options as necessary, but for this guide we will leave the defaults. It is important, however, that you select to Use Existing Resource Group and specify the group you created in the last step. You’ll also want to select the same Location as you did before. Azure will not deploy VMs (or other objects) if the Location doesn’t match logically between the various components that the object will consume. Click Create.

CreateAppRegistrationDetails
Now you need to set up an Azure Active Directory application so that vRA can authenticate. Locate the Active Directory icon on the left hand side and click it. Next, click App Registrations and select Add. The most astute readers will notice that there are certain parts of some of my screenshots deleted – sorry about that! Had to remove sensitive information.

Enter a Name for your AD App – it can be anything you like, as long as it complies with the name validation. Leave Web app/API as the Application Type. The Sign-on URL is not really important for the purposes of this configuration – you can enter really anything you want here. In this example, we are using a dummy vRA 7 URL. Click Create (not pictured above, but you should have the hang of it by now!)

SetupADAppSorry the above image is a little squashed. You can always click them for larger resolution!

Now you need to create a secret key to authenticate to the AD Application with. Click on the name of your new AD Application (in this case vRADevTest) at the left. Make sure you note down the Application ID for later. Then, select the All Settings button in the next pane. Choose Keys from the settings list.

CreateAppKey
Now, enter a Description for your new key and choose a Duration. Once you have entered those, click Save in the upper left of the blade – but note the warning! You will not ever get another chance to retrieve this value. Save the Key Value for later.

ConfigureAPIPermissions
Now, look back to the left and select the Required Permissions option for the AD App. Click Add to create a new permission.

SelectAzureSMAPI
Click Select an API and choose the Windows Azure Service Management API, then click Select

AssignSMAPIPermissions
Click the Select Permissions step at the left, then tick the box for Access Azure Service Management as organization users (preview) – then click Select. Once you do this, the Done button on the left will highlight. Click that as well.

There’s one final step in the Azure portal. Now that the AD Application has been created, you need to authorize it to connect to your Azure Subscription to deploy and manage VMs!

BackToSubscriptionsView
Click back on the Subscriptions icon (the Key) and select your new subscription. You may have to click on the text of the name to get the panel to slide over. Select the Access control (IAM) option to see the permissions to your subscription. Click Add at the top.

SelectRole1
Click Select a Role and choose Contributor from the list

SelectUsers
Click the Add Users option and search for the name of your new AD Application. When you see it in the list, tick the box and click Select, then OK in the first blade.

RolesAssigned
Repeat this process so that your new AD Application has the Owner, Contributor, and Reader roles. It should look like this when you’re done.

Part 2 – Azure CLI and Other Setup

To do the next steps, you will need the Azure CLI tools installed. These are freely available from Microsoft for both Windows and Mac. I won’t go into great detail on how to download and install a client application here – but you can get all the info you need at https://docs.microsoft.com/en-us/azure/xplat-cli-install. For the purposes of this guide, please remember that I use a Mac.

AzureLoginStep1
Once you have the Azure CLI installed, you will need to authenticate to your new subscription. Open a Terminal window and enter ‘azure login’. You will be given a URL and a shortcode to allow you to authenticate. Open the URL in your browser and follow these instructions to authenticate your subscription.

EnterAuthCodeStep1
Enter your Auth Code and click Continue

EnterAuthCodeStep2
Select and log in to your Azure account…

AuthSuccessWeb

AuthSuccessCLI
And if all went well, you now have a success message in both your browser and the CLI. Nice work!

AzureAccountSet
If you have multiple subscriptions, as I do, you’ll need to ensure that the correct one is selected. You can do that with the ‘azure account set <subscription-name>’ command. Be sure to escape any spaces!

RegisterComputeProvider
Before you go any further, you need to register the Microsoft.Compute provider to your new Azure subscription. This only needs to be done once, which means it’s easy to forget! The command is just ‘azure provider register microsoft.compute’ – and it has timed out the first time in 100% of my test cases. So I left that Big Scary Error in the screenshot for you – don’t worry, just run it a second time and it will complete.

AzureVMImageList
Now, let’s use the Azure CLI to retrieve an example VM image name. These will be used in the vRA Blueprints to specify which type of VM you’d like to deploy. To do this, you’ll use the ‘azure vm image list’ command. In my example, the full command was ‘azure vm image list –location “West US” –publisher canonical –offer ubuntuserver –sku 16.04.0-LTS’  – this limits the list of displayed options to only those present in my West US location, published by Canonical, of type Ubuntu Server, containing the string 16.04.0-LTS in their name.

Choose one of these images and record the URN provided for it. As an example: canonical:ubuntuserver:16.04.0-LTS:16.04.201611150

So, to recap – you have set up your Azure subscription and should have the following list of items recorded:

  • Subscription ID
  • Tenant ID
  • Storage Account Name
  • Resource Group Name
  • Location
  • Virtual Network Name
  • Client Application ID
  • Client Application Secret Key
  • VM Image URN

Now, let’s move on to actually configuring vRA!

Part 3 – Configuring vRA

This section assumes that you have already deployed vRA with the default tenant, have created your basic users and permissions, and have at least one business group ready. This basic level of vRA setup is outside the scope of this guide.

AdministrationTab
Once you are logged in as an Infrastructure/IaaS administrator, proceed to the Administration tab and select vRO Configuration from the menu at the left (not pictured.) Then, choose Endpoints and select New to set up a new endpoint.

The Azure endpoint is not configured from the traditional Infrastructure tab location because it is not managed by the IaaS engine of vRA – it is presented via vRO and XaaS.

SelectAzureType
Select the Azure plug-in type and click Next

AzureEndpointName
Enter a Name for your Endpoint and click Next again

EnterAzureSubscriptionDetails
Now the fun part! Remember all that info you copied down earlier? Time to use it! Fill in the Connection Settings with the details from the subscription configuration you did earlier. You won’t need to change the Azure Services URI or the Login URL, and the Proxy Host/Port are optional unless you know you need one.

Click Finish and the connection should be created!

FabricGroups
Next, navigate to the Infrastructure tab and select Endpoints (not pictured,) followed by Fabric Groups. In this example I don’t yet have a Fabric Group, so I will create one by clicking New.

NewFabricGroup
Remember a little while ago that I mentioned the Azure Endpoint is not managed by IaaS – so you won’t need to select any Compute Resources here. You just need to ensure that your user account is a Fabric Administrator to continue the rest of the configuration. If you already have this right, you may skip this step.

Now, refresh the vRA UI so that your new Fabric Administrator permissions take effect.

CreateNewReservation
Once that’s done, navigate to the Infrastructure tab and the Reservations menu. Select the New button and choose a reservation of type Azure.

NewReservationGeneral
Fill in a Name and select a Business Group and Priority for the reservation, then click on the Resources tab

NewReservationResources
Enter your Subscription ID – be sure this is the same subscription ID that was specified in your Endpoint configuration. Requiring this field allows the mapping of many reservations to many endpoints/subscriptions.

Then, add the Resource Group and Storage Account which you created earlier. This is not required, but it does save some steps when creating the Blueprint later.

Click on the Network tab.

NewReservationNetwork
Enter the name of the Virtual Network you created earlier. Also note that you can set up Load Balancers and Security Groups here. Click OK to save the reservation.

CreateMachinePrefix
Next, you’ll need a Machine Naming Prefix. Click on the <Infrastructure menu option (not pictured) and then select Administration (also not pictured) and finally Machine Prefixes. Enter a string, number of digits and next number that works for you – I used AzureDev-### starting with the number 0. Be sure to click the Green Check to save the prefix.

This prefix will be applied to any objects provisioned in a request – whether they are VMs, NICs, storage disks, etc. This helps the grouped objects to be easily located in an often busy Azure environment.

BusinessGroups
Now, click the Administration tab, followed by the Users and Groups menu (not pictured) and the Business Groups option. Select the business group that you plan to deploy with – in this example I have three to choose from and will be using Development.

ChooseBGMachinePrefix
Select your new Default Machine Prefix and click Finish.

Part 4 – Building a Blueprint

Now that the groundwork is laid, let’s build, entitle, and deploy a simple Azure blueprint!

DesignTab
Head over to the Design tab and make sure the Blueprints menu is open. It should be the default. Click New to begin designing a blueprint.

BlueprintProperties
Give your blueprint a Name and click OK

BlueprintCanvas
Ensure the Machine Types category is selected and drag an Azure Machine to the canvas. Increase the Maximum Instances to 3 – this will make your Azure machine scalable! Click the Build Information tab to proceed.

BuildInformation
Now you can begin filling out details about the machine itself. Select a Location – or one will be chosen for you from the reservation. You can also choose a Naming Prefix or allow the one you set up a moment ago to be the default. You can choose to select a Stock VM Image and paste the URN you retrieved from the Azure CLI, or you can specify a custom, user created one. Here you can also specify the Authentication options as well as the Instance Size configuration. If any of these options are left blank, they will be required at request time.

Note that when editing a field, you will see an editing dialog appear on the right of the blueprint form. This is to allow you additional flexibility in the configuration; please be sure to click ‘Apply‘ to save any changes. Also note that there are many helpful tooltips throughout the blueprint designer to help you along.

Click the Machine Resources tab to move on.

MachineResources
Here you can specify your Resource Group and Availability Set – and as before, you can fill in the one you created manually or allow vRA to create new ones for you. Remember to fill in the information on the right hand side and click Apply to save the values!

Click Storage to move to the next step.

MachineStorage
The Storage tab allows you to specify details about your machine’s storage capabilities. You can specify the Storage Account here if you choose – or it can be inherited from the Reservation. If you explore this tab, you’ll see you can also create additional data disks as well as enable/disable the boot diagnostics functionality. For this example we will just create a simple OS disk configuration.

Now, click on the Network tab.

MachineNetwork
This is where you can configure advanced networking capabilities. In this example, you won’t fill anything in and we will instead allow the Azure reservation to apply the networking properties you specified earlier. Click Finish to save your blueprint.

PublishBlueprint
Select your new blueprint and Publish it.

Now you must entitle your new blueprint. Because the steps to complete this operation can be highly dependent on the environment you’re doing it in, we will skip the details on how to create an entitlement and add this blueprint to it. Let’s move right ahead to provisioning the VM!

Part 5 – Deploying a Blueprint

I hope you’re glad you stuck with me this far! To recap, so far you have:

  • Created and configured your Azure subscription for vRA
  • Collected up a list of all the important pieces of data needed to provision to Azure
  • Configured vRA to deploy to Azure
  • Built your first Azure blueprint

There’s just one thing left to do….

vRACatalog
Navigate to the Catalog tab, locate your new Azure blueprint and click Request.

RequestDetails
Feel free to click around the request details – you’ll see that anything you specified in the blueprint itself is now a locked field. Other fields are still open and available for editing. You can create some seriously flexible requests by locking and unlocking only specific fields – the form is highly customizable.

When you’re done exploring, click Submit!

vRARequestStatusSuccessful
You can monitor the status of the request as you normally would, in the Requests tab.

vRADeployments
After the provisioning completes, you’ll be able to see your new Azure VM in vRA…

AzureProvisioned
…as well as in the Azure portal itself! You can see that the Naming Prefix was applied to both the VM and the vNIC that was created to support it.

SouthernTierPumking
This post was brought to you courtesy of Southern Tier Brewing’s Pumking – possibly the only good pumpkin beer ever. It hits all the natural squash and spice notes without ever feeling extracted, artificial, or overwhelming. And it gets bonus points for being from my home town. Yum!

I hope this guide has been helpful and that you’re as excited as I am about this great new addition to vRealize Automation’s repertoire. Please leave any feedback in the comments, and don’t forget to follow me on Twitter!

vRealize Automation 7 Management Pack for vRealize Operations

If you’re an SDDC administrator, you probably already know about the power and operational visibility that vRealize Operations brings to your environment. With the newly-released vRealize Automation 7 Management Pack for vRealize Operations, that operational visibility can be extended to be tenant-aware and help monitor your vRA environment in a whole new way.

This new Management Pack gives you comprehensive visibility into both performance and capacity metrics of a vRA tenant’s business groups and underlying cloud infrastructure. By combining these new metrics with the custom dashboarding capabilities of vRealize Operations, you gain an unprecedented level of flexibility and insight when monitoring these complex environments.

The purpose of this post is to walk you through the implementation of this new Management Pack – so, let’s get right to it.

You can download the Management Pack from the VMware Solution Exchange here.

Part 1: Enabling vROps as your Metrics Provider

First, let’s review what you’ll see before you integrate vRA and vROps. Looking at the details of any deployed item, you can see the highlighted white space – space that can definitely be put to more productive use.

Item_Details_Before_Integration

Assuming you’re logged in as a vRA Tenant Administrator, click on the Administration tab, then the Reclamation button in the menu at the left. Select Metrics Provider and you’ll see the configuration panel for the vROps endpoint. Fill in the appropriate details for your vROps instance and click Test Connection. Once it succeeds, click Save.

Set_Up_Metrics_Provider

You will probably be prompted to accept the SSL certificate offered up by your vROps instance. Click OK to accept the certificate, provided you trust it!

Accept_vROps_Cert

Now, if you click on the Tenant Machines option to the left, you’ll be presented with a list of all of your provisioned machines. You can see that now there’s a Health status badge for each machine. In my case, the Health is reporting an “Immediate” (orange) status for many of my virtual machines, due to very heavy utilization in my lab. You can also see the average CPU, Memory and Network consumption for each machine – data pulled directly from vROps. This consumption data can be used directly from within this view to initiate reclamation requests. For example, if a VM was identified here as idle, the VM owner could be notified and the resources recovered.

Tenant_Machines_View

Click back to the Items tab and view the same object you looked at earlier. You will see that the white space now contains a vROps-driven Health badge, with information about any possible issues. When you’re ready, log out of your vRA instance.

Item_Details_After_Integration

Part 2: Configuring vRealize Automation

You’ll need to log in as  the default administrator for this next step – administrator@vsphere.local

Log_In_vRA_Default_Administrator

Click on the Administration tab, followed by the Tenants menu button at the left. Locate the Tenant that you plan to link vROps to and Edit it. In this example, I am modifying the vsphere.local Tenant.

Locate_Target_Tenant

Now, select the Local Users tab. Click the +New button to add a new user and fill in the requested details. In this case, my new username is “vropsmp” – and since we are creating this local user in the vsphere.local tenant, the full account is “vropsmp@vsphere.local“. Click OK and then Next.

Add_New_Local_User

This will place you on the Administrators tab. Using the Search boxes, find and add your new local account to both the Tenant Administrators and the IaaS Administrators role. Click Finish when you’re done, and then Log Out of vRA.

Assign_Tenant_Rights

Now you’ll need to log back in as your normal vRA Tenant Administrator to finish the configuration.

Log_In_vRA_Tenant_Admin

Click the Infrastructure tab, then Endpoints from the menu on the left. Select Fabric Groups from the sub-menu and then click to edit your Fabric Group. In this example, the Fabric Group is named Dev Cluster.

Navigate_To_Fabric_Groups

Search for and add your new local user to the list of Fabric Administrators. Remember, in this example the user is named vropsmp@vsphere.local. Click OK to save the Fabric Group.

Edit_Fabric_Group

Now, click on the Administration tab, followed by Users & Groups from  the menu on the left. Select the Directory Users and Groups sub-menu and search for your new local user. Click the user’s name to edit it.

Navigate_To_Directory_Users

In the list to the right titled “Add roles to this User“, scroll down until you find the Software Architect role. Select it and then click Finish to save the account.

Assign_Software_Architect_Role

Part 3: Configuring vRealize Operations

Once you’ve downloaded the new Management Pack (again, found here) you’ll need to import it into vROps and configure it to retrieve data from vRA.

Log in to vROps with an administrative user account.

Log_In_vROps

Click on the Administration tab, and ensure Solutions is selected. Click the + symbol to import a new Solution.

Import_New_Solution

Click the Browse button to select the downloaded Management Pack, then click Upload.

NOTE! If you already had the earlier vROps Management Pack for vRA installed, you may have to do a “force install” by selecting the first checkbox. This is because the version number scheme was changed, and vROps recognizes the NEW MP as being an OLDER version. This is normal, if a bit cumbersome.

Click Next when the upload is verified and you are ready to proceed.

Upload_New_Solution

Accept the EULA (after reading it carefully first, of course) and click Next again.

Accept_EULA

The installation will run for a while. When it shows “Completed”, click Finish.

Complete_Installation

Locate the new Management Pack in the list of Solutions and highlight it. Click the Configure icon (gears) to bring up the configuration dialog. Fill in a Display Name and Description as well as your vRA URL and the name of the Tenant you want to connect to. In this example, the Tenant is vsphere.local. Click the + sign to start setting up credentials next.

Configure_Solution_Basics

Fill in the credential details as shown – your SysAdmin should be the administrator@vsphere.local administrative account, and your SuperUser will be the local user you created at the beginning of these steps. In this example, that local user is vropsmp@vsphere.local. Click OK when you’re done.

Manage_Credential

Click the ‘Test Connection‘ button. You’ll be prompted with two SSL certificate dialogs – accept them both, if you trust the certificates. You see two because the Management Pack is communicating with both your core vRA appliance as well as your IaaS server(s).

Test_Connection_Accept_Cert_1

Accept_Cert_2

If you’ve set everything up properly, you’ll see a message like this one. Click OK.

Test_Successful

Click on Save Settings to save your adapter configuration. You’ll be prompted with a “Save Successful” dialog – click OK here as well – then click Close.

Save_Solution_Settings

If everything’s gone according to plan, you should now see that your Management Pack is configured and receiving data from your vRA instance.

Solution_Details

Part 4: Reviewing Dashboards

Now that all of the configuration is complete, you’re ready to start consuming the rich data exposed by your new integration. Click on the Home tab in vROps, followed by the drop-down arrow for the Dashboard List. Hover over the vRealize Automation sub-menu to see the 4 available default  dashboards.

Navigate_To_vRA_Dashboards

The vRealize Automation Overview dashboard shows information about the entire vRA instance – including component health and a whole host of metrics about each individual component of the instance. This is useful for troubleshooting and analyzing performance across your entire implementation of the vRA stack.

vRealize_Automation_Overview

The vR Automation Tenant Overview dashboard provides exactly that – an overview of the various risk and health metrics pertinent to each configured vRA Tenant.

vR_Automation_Tenant_Overview

The vR Automation Cloud Infrastructure Monitoring dashboard allows you to see what impact infrastructure issues are having on tenant virtual machines, and what outstanding alerts may be present for those machines and infrastructure.

vR_Automation_Cloud_Infrastructure_Monitoring

Finally, the vR Automation Top-N Dashboard provides highlight Top-N metrics, such as the most popular Blueprints, most wasteful Tenants, the Business Group with the most alerts, etc.

vR_Automation_Top-N_Dashboard

And, of course, all of the objects which are exposed by the Management Pack can be viewed in the vRealize Automation Environment view. These objects can all be referenced by Super Metrics, or custom dashboards, or scheduled reports – but those are all beyond the scope of this guide.

vRealize_Automation_Environment

That just about wraps it up – except, of course, for the most important part…

This post was brought to you by New Helvetia Brewing Company’s Mystery Airship 2.0 Imperial Chocolate Porter, brewed with Ginger Elizabeth’s Oaxacan Spicy Chocolate. This is quite possibly the single greatest beer I have ever tried – the darkness of the porter is supplemented by the brightness of the ginger and creamy feel of the chocolate. The flavors dance on your palate and then vanish in a fog of lingering, dark spice. I honestly think I found my desert island beer!

New_Helvetia_Ginger_Elizabeth_Porter

Happy Automating!

vIDM Attribute Mapping in vRA 7

It seems like the more time I spend with the new VMware Identity Manager (vIDM) in vRealize Automation 7, the more great new capabilities I discover. Today’s post comes directly from a customer request, and discusses how to use vIDM Attribute Mapping in vRA 7.

Due to complexities in this customer’s Active Directory environment, they have the “email” attribute in their user accounts populated – but it does not contain the user’s actual email address. This means that vRA is unable to send them notifications, as it automatically inherits this field and uses the information therein.

Have no fear, vIDM is here.

Here you can see my account with the default configuration. My email address is set to jon@corp.local, but that just isn’t where I receive my email.

User_Account_Default_EMail

Looking at my account inside of Active Directory, we can see that this address is set in the ‘E-mail’ field, which maps to the ‘mail‘ attribute in LDAP.

User_AD_Email

But, if we look at the Attribute Editor, we can see that the LDAP ‘otherMailbox‘ attribute contains my preferred email address of ‘jon@vaficionado.com

User_AD_otherMailbox

So, how can I change my vRA configuration to utilize that otherMailbox attribute instead? It’s very easy. Start by clicking on the Administration tab in vRA. Then select the Directories button on the left hand side and edit your Active Directory shown to the right.

Navigate_To_Directory_Configuration

Next, you’ll be presented with the Active Directory settings page. Click on the Sync Settings button.

Sync_Settings

Here you’ll see a whole host of advanced synchronization options that you can change. Click on the Mapped Attributes button at the top, then select the dropdown next to email. Select Enter Custom Input… from the menu.

Mapped_Attributes

Now, enter the new Active Directory attribute name that you want to retrieve the email address from. In this example, the new attribute is named otherMailbox. Click Save & Sync to save your settings and update the user accounts.

Enter_New_Attribute_Name

You’ll now be given the opportunity to review the proposed changes once the sync is completed. You can see in this example, there are 4 AD accounts that will have their attribute mappings updated. Click Sync Directory.

Review_Sync_Settings

Once the sync is completed (this may take some time, depending on how many objects were being updated and the size of your AD, etc) go back to the Administration > Directory Users and Groups view and find your user account again. You’ll notice that the email address has now been updated to reflect the contents of the preferred attribute.

Updated_EMail_Address

Pretty cool, huh?

This post was brought to you by Terrapin Beer Co’s Poivre Potion, a very unique dry-hopped pink peppercorn Saison. I love the way the spicy and sweet notes of the peppercorns play off the bitterness of the hops and the farmhouse funk of the Saison yeast strains. Delicious and easy to drink.

Terrapin_Poivre_Potion

Happy automating!

Configuring vRA 7 for 2 Factor Authentication

 

One of the most exciting new features in vRealize Automation 7 is the addition of the VMware Identity Manager (or vIDM) to act as the identity provider. This brings a whole host of new capabilities, but one of the key among them is the addition of simple and flexible multi-factor authentication. This  guide will walk you through the process of configuring vRA 7 for 2 factor authentication, using Google Authenticator as our example token.

In this example scenario, a vRA 7 environment is already set up and fully functional using traditional username and password authentication. The guide also assumes you have a basic CentOS server set up and available for configuration. Both of these steps are outside the scope of the instructions below.

Part 1: Configuring the Linux Host

First, ensure that you have proper DNS resolution set up for the CentOS host. This host will act as your authentication intermediary, processing both the Active Directory username and passwords as well as the Google Authenticator token. Since AD is involved, DNS and time configuration will be critical.

Create_DNS_Record

Next, SSH to your Linux host. You’ll need to be a privileged user (i.e. root) for these operations, so that’s what I’ve logged in as here. Your configuration may require you to log in as a standard user and then su to root, or use sudo.

Log_In_Host_SSH

Next, you must edit the SELinux configuration.

vi /etc/selinux/config

SELINUX=disabled

Ensure that the SELinux policy is set to either permissive or disabled as shown below. While it is definitely possible (and probably advisable) to keep SELinux enabled for this configuration, the additional steps to do so are outside the scope of this guide.

Edit_SELinux_Configuration

Now, remember what I said about DNS and time being critical when integrating with Active Directory? You’ll need to set up NTP services on your host to ensure that there’s no time drift for authentication to work properly.

yum install ntp -y
ntpdate <your_ntp_server_here>

This will install the ntpdate package on your host, as well as set the time source. Ensure that you set this NTP server to the same one that provides time to your Domain Controllers – this will prevent time related authentication failures.

Install_Configure_NTP

Now would also be a great time to confirm your DNS configuration is correct. Check the /etc/hosts file and ensure that your hostname is mapped to the correct IP address. Also check your /etc/resolv.conf file to ensure that your host is pointing at a DNS server which can properly resolve your Active Directory. In the example shown, the DNS server is set directly to our Domain Controller.

Confirm_DNS_Config

Next, ensure that the wget package is installed. Your host may already have this installed.

yum install wget -y

Install_wget

Great. The basic system is now set up and ready to start loading the software that will do the heavy lifting.

First up will be the PowerBroker Identity Suite, or PBIS. These utilities will enable the simple addition of AD authentication to your Linux host. The commands below will add the PBIS RPM repository so that yum can download and install the packages.

rpm --import http://repo.pbis.beyondtrust.com/yum/RPM-GPG-KEY-pbis
wget -O /etc/yum.repos.d/pbiso.repo http://repo.pbis.beyondtrust.com/yum/pbiso.repo
sed -i "s/mirrorlist=https/mirrorlist=http/" /etc/yum.repos.d/epel.repo
yum clean all
yum install pbis-open -y

Install_PBIS

Now that PBIS is installed, you can join your Linux host to your AD domain.

domainjoin-cli join corp.local <your_domain_username>

This command will actually join the host to the domain, creating a computer object and all the required Linux PAM configuration. Ensure you use a username with the rights to add machines to the domain – in the example here, we used the default Administrator account.

/opt/pbis/bin/config AssumeDefaultDomain true

Next, this command will ensure that the domain you joined is always assumed to be the default. This saves you entering DOMAIN\username notation for everything you do.

Join_AD_Domain

Now open up your Active Directory Users and Computers snap-in. By browsing to the default Computers container, you can validate that your Linux host is now added to the Active Directory. In this example, it is named util-01a and is listed as a CentOS 6.3 host running PBIS Open 8.3.

Validate_Domain_Membership

And while you’re in the ADUC view, create a domain group called RADIUS_Logon_Disabled in the Users container. You don’t need to add any users to it now – this will be used only if you want to deny any users the ability to authenticate against RADIUS without completely disabling their account. We’ll come back to this group later.

Create_Denied_Logon_Group

Now, reboot your Linux host. This ensures that all of the PBIS configuration is in full effect.

Once the host is fully restarted, log in as the same privileged account you were using before. We’re not done yet!

wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
rpm -Uvh rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm

This will enable another external repository, so that you can obtain the QR code generator that will be used with Google Authenticator…

Prepare_RPMforge_Repo

yum install qrencode qrencode-devel git pam-devel gcc -y

…and this will grab and install all the packages and dependencies needed to build the Google Authenticator components.

Install_Build_Tools

Since the Google Authenticator utilities aren’t delivered as an RPM package, they’ll need to be built from source. To do that, you’ll download the source files from a Git repository and compile them directly on the Linux host. Don’t worry, this sounds a lot harder than it is.

cd /root
git clone https://code.google.com/p/google-authenticator
cd /root/google-authenticator/libpam
make && make install

This checks out the latest version of the Google Authenticator code, downloads it to your local system, compiles it and installs it. Easy, right?

Download_Compile_Google_Authenticator

This is the last package installation, honest. Download and install the FreeRADIUS server, using:

yum install freeradius -y

Install_FreeRADIUS

Now that all the packages are installed, there’s some configuration to be done. This part requires some config file editing, so be sure you’ve got your editor of choice handy and read the steps carefully – small mistakes can have big impact here!

First, the user which FreeRADIUS runs as must be changed. By default, the server executes as the radiusd user, but because we will need to read Google Authenticator tokens from every user’s home directory, it is far easier to run the service as root instead. There are of course other ways to make this possible without running as root, but they are outside the scope of this guide. In a production environment, you should definitely explore doing so.

vi /etc/raddb/radius.conf

Change:
user = radiusd
group = radiusd

To:
user = root
group = root

Change_FreeRADIUS_User_Group

Next, edit the radiusd.conf file to deny access to members of the AD group you created earlier. This is accomplished by adding the text shown here, in the “Deny access for a group of users” section of the file.

vi /etc/raddb/users

Add:
DEFAULT Group == "RADIUS_Login_Disabled", Auth-Type := Reject
 Reply-Message = "Your account has been disabled."

DEFAULT Auth-Type := PAM

Configure_Denied_RADIUS_Users

Now, FreeRADIUS must be configured to accept PAM-based authentication. PAM is the Linux Pluggable Authentication Module framework, and is what makes all of this fancy authentication possible.

vi /etc/raddb/sites-enabled/default

Uncomment the line shown so it just reads "pam"

Enable_RADIUS_PAM

Once the FreeRADIUS server is configured to accept PAM authentication, PAM itself must be configured to use the correct mechanisms, in this case combining Active Directory authentication with Google Authenticator tokens. To do this, edit the /etc/pam.d/radiusd file and comment out all of the existing lines, then add the configuration below

vi /etc/pam.d/radiusd

Comment all existing lines by prefixing with #

Add:
auth requisite pam_google_authenticator.so forward_pass
account required pam_lsass.so use_first_pass

Configure_PAM_Modules

Finally, the RADIUS server must be configured to authenticate the vRealize Automation server itself. This is done by pairing a shared secret with the hostname of the system. Edit the /etc/raddb/clients.conf file and add the text specified in the section shown. Be sure not to add this new client definition inside the default definition. In the example shown, the client is vra-01a.corp.local, the secret is VMware1!, and the shortname is vra-01a. Fill in your specific details instead.

vi /etc/raddb/clients.conf

Add:
client <your-full-vRA-VA-hostname> {
 secret = <your-shared-secret>
 shortname = <your-vRA-VA-friendly-name>
}

Add_RADIUS_Site

Now, start the FreeRADIUS server.

service radiusd restart

Restart_radiusd

Part 2: Configuring vRealize Automation

Now that the Linux host is configured to process the authentication requests, you’ll need to configure vRealize Automation’s VMware Identity Manager instance to leverage it.

Log in to vRealize Automation as a Tenant Administrator.

Click on the Administration tab, then the Directories Management button on the left. Select the Connectors button and you’ll see the screen pictured. This is where you’ll configure the vIDM Directory connection. Click on first.connector as shown.

Configure_vRA_Directory_Connector

You’ll be presented with the screen below. Click on the Auth Adapters button. Notice that the RadiusAuthAdapter is Disabled. Let’s change that – click on RadiusAuthAdapter.

Configure_Auth_Adapters

Here you’ll see the configuration for the vIDM RADIUS adapter. Fill in the fields as shown, substituting  the correct Radius server hostname/addressShared Secret (remember you entered this in an earlier step – in this  example it was VMware1!) and Realm prefix. The Realm prefix is your domain name, with the trailing slash character. Also, note the Login page passphrase hint has been customized – this reminder will display on the login page to help guide users to enter the correct data.

Do not enable the Secondary Server at this time – leave all the rest of the fields as-is.

Click the Save button at the bottom of the screen, then switch back to the vRealize Automation tab in your browser.

Configure_RADIUS_Adapter

Now, click on the Network Ranges button and select Add Network Range. This will allow you to specify groups of IP addresses which will use a particular authentication config. It’s a good idea to configure just one or two IPs for testing purposes initially, so that you don’t accidentally lock yourself out of the environment.

vRA_Network_Ranges

The Network Range configuration is pretty straightforward. Just enter a name, description and IP range. Make the starting and ending IP addresses the same to specify only a single host. In this example, we are limiting this range to the local desktop. Click Save.

Add_Network_Range

Click on Policies  to the left and then edit the default_access_policy_set. Remember that you can create multiple policies for multiple scenarios.

vRA_Access_Policies

Click on the Green + sign to add a new policy rule.

Edit_Default_Policy

Configure the policy rule as shown:

  • If a user’s network range is: <your new network range here>
  • And the user is trying to access content from: All device types
  • Then the user must authenticate using: Radius Only

Click Save.

Add_Policy_Rule

Now, grab the icon highlighted in red with your mouse and drag the new rule to the top of the list. Click Save.

Reorder_Policy_Rules

vRealize Automation is now configured to use RADIUS authentication, combining both Active Directory credentials with a Google Authenticator token.

Part 3: Enabling Users for Google Authenticator

Now that the Linux host has been built and configured and vRA has been set up to take advantage of it, you need to create tokens for your users.

Re-connect to your Linux host from Part 1 using SSH, or if you still have an active session simply switch back to it. You should be authenticated as root at this point, as you will be assuming the identity of your AD users to create their tokens.

In the pictured example, we are becoming a user named mary. Mary is an AD user who has never before logged in to this Linux host – yet we were able to assume her identity by authenticating against Active Directory. Pretty cool! You can also check that you are indeed logged in as Mary by running whoami.

su mary
whoami

Su_to_AD_User

Now, you’ll create the Google Authenticator token. This can be done by running the following command:

google-authenticator -tdf -r 3 -R 30 -w 17 -Q UTF8

Notice that the command creates a huge QR code, a Secret Key, and 5 emergency scratch codes. These codes can be used in the event that you don’t have your smart device handy, but each can only be used once. Keep those in a safe place. The QR code is a graphical representation of the alphanumeric secret key printed directly beneath it.

Create_Google_Authenticator_Token

Here’s the fun part. Pick up your nearest handy smart device. It could be a smartphone, a tablet, etc. I use an iPhone, so the following images were captured there.

Search for the Google Authenticator app in the App Store, or Google Play, etc. Download it – it’s free.  Open the app.

Google_Authenticator_App_Store

Tap the “Scan Barcode” option and grant the application access to your camera. You can also select Manual Entry and type in the alphanumeric secret key – but where’s the fun in that?

Set_Up_Google_Authenticator

Point your phone at the QR code generated on the screen and the app will do the rest!

Scan_QR_Code

You can see that the Authenticator app has automatically generated a token for Mary, showing her name and the server which she’s authorized for. The little Pac-Man thing to the right is a timer – these tokens are only good for a single use, and only valid for 30 seconds.

View_User_Token

Part 4: Testing Your Work

To recap, you’ve just:

  • Built a Linux host to handle the task of authenticating against Active Directory and Google Authenticator
  • Configured vRealize Automation to leverage that host as an authentication source
  • Created a time-based token for one of your Active Directory users

Now it’s time to put it all together and test the configuration.

Open a new browser window. I find that an ‘Incognito’ or ‘Private Browsing’ window works best, since you probably have another window logged in as your Tenant Administrator already. Notice that you are now prompted for the username and AD Password + Google Auth Code – that was the free text you entered a few steps back to help guide your users.

Login_vRA

Log in using the new token on your smartphone. Assuming the following parameters:

  • Username = Mary
  • Password (AD) = VMware1!
  • Google Authenticator Code (from smart device) = 098765

You would log in with a username of “mary” and a passcode of “VMware1!098765

Login_Successful

Voila!

This post was brought to you by Breakside Brewery’s Salted Caramel Stout, which was genuinely instrumental in getting me through developing this configuration. Notes of chocolate play on the nose while sea salt and fresh caramel round out the palate in one of the smoothest, most pleasant stouts I’ve ever tried.

Breakside_Salted_Caramel

Happy automating!

Special thanks to Ed Kaczmarek for contributing to this guide – follow him at @edkaczmarek!

Reflecting on Hands-On Labs at VMworld 2015

Now that I’ve had a day or two to decompress after another action-packed VMworld, I thought it would be appropriate to just post a few thoughts about the experience.

I became involved with the Hands-On Labs shortly before VMworld 2014, making this my second cycle with the program. At the time, I had no idea how difficult or how rewarding the experience would be. As it turns out, participating in the Labs has been one of the single most personally and professionally satisfying undertakings of my life.

The development cycle began back in February of 2015, when a few of my fellow captains and I began developing what would be known as the “SDDC Base Pod” – a fully integrated single-site environment based on vSphere 6.0. This pod would contain all of the necessary components to showcase VMware’s Software-Defined Datacenter. Once extensive performance and integration testing had been completed, the pod was saved and made available to the rest of the individual lab development teams. This happened around May – and is when we really began creating our lab-specific content. All in all, each of us has contributed 500+ hours to the development, testing and delivery of this lab.

Working with Kim (@KCDAutomate), Shawn (@ShawnMKelly) and Grant (@GrantOrchard) with Burke (@TechnicalValues) as our leader, we laid down the additional software components, configuration, development and documentation to create the 8 amazing modules which comprised our 2015 lab. I’m pleased to be able to reveal the details of the lab now that VMworld has concluded:

HOL-SDC-1632 – vRealize Automation Advanced: Integration and Extensibility

A list of the modules is as follows:

  • Module 1 – You Need More Integration
  • Module 2 – An Introduction to Extensibility
  • Module 3 – Integrating vRealize Automation with the VMware Cloud Management Platform
  • Module 4 – Integrating vRealize Automation with Infoblox IPAM
  • Module 5 – Integrating vRealize Automation with Puppet Enterprise
  • Module 6 – Integrating vRealize Automation with NSX
  • Module 7 – XaaS Services with Advanced Service Designer and vRealize Orchestrator
  • Module 8 – Working with the vRealize Automation API

Each of the above were lovingly handcrafted by our team to show off not only the power and flexibility of the vRealize Automation engine, but also the amazing ways that it can be integrated into the other components of the VMware Cloud Management Platform as well as third party solutions that might already exist in your infrastructure.

But creating the labs are only the start. Delivering the content at both VMworld events and supporting it throughout the year is when the real work begins. The amazing Hands-On Lab staff works tirelessly to make sure that every attendee and lab user has a seamless, enlightening, engaging and enthralling experience. There are core staff, support staff, principals, captains, proctors and administrators. All of them play a role in making sure that the premier hands-on learning event in the industry can be a reality, and they all deserve huge thanks for their roles.

According to the surveys we received, our lab was a resounding success – as were the Expert-Led Workshops we hosted to teach our customers all about extensibility.

But, of course, events like this can’t be all work. We have plenty of fun too – and I’m very pleased to be able to call so many of these rockstars my friends, and want to thank some of them. Particularly:

  • Jad, Chris and Tina for wrangling all the staff and dealing with all the administrative work that’s so important with this many staff
  • Kim, Grant, Shawn and Burke for being the most amazing team I can think of. We’ve helped each other learn and grow so much in such a short time, and it’s been incredible
  • Doug, Bill and Dave for supporting us as we built, tested, reimagined, rebuilt, re-tested and rebuilt the environments time and time again
  • The rest of the principals, captains and proctors who helped create all the other content and made the lab room the bustling hive of expert conversation it was

That’s all for now – we’ll see some of you in a few weeks at VMworld in Barcelona – and keep an eye on the Hands-On Labs portal for this year’s content to be available to you at home!

Now if you’ll excuse me, my grill is hot and these rib-eyes are calling my name. Paired with a 2011 Miner Oakville Cabernet, I don’t think I can wait much longer.

Miner_Oakville_2011_Cab_And_Rib_Eyes

Creating new vRA Workloads in a specific AD OU

This week, I’ve had several customers individually approach me with this question – how can they specify the OU which a Windows VM should land in when it’s created via vRA?

This is a great question and a very important operational task to accomplish – OU membership determines so much vital configuration for a Windows machine.

It seems like most of these customers have a tendency to assume they are going to create the VM first, and relocate it to a new OU later. But there’s a much more streamlined way to do it. By binding a workflow to the IaaS BuildingMachine lifecycle stage, you can pre-stage the computer object in AD before it’s even provisioned. That way, when it first adds to the domain it will already be present in the correct OU. This also has the added benefit of ensuring all group policies are inherited right away, rather than requiring additional reboots.

I’ve put together a quick example here that should help you see how to do just this.

To use the example workflow attached above, you must already have your vRO instance registered with vRA and the extensibility customizations installed. We also assume that you have correctly configured the Active Directory plugin, and that the example vRA blueprint you will deploy has a vSphere Customization Specification attached which adds the VM to your AD domain.

First, import the workflow into a vRO folder of your choosing.

Then, browse to it in the workflow tree and select it. On the General tab, you can see there are two Attributes that must be updated for your own envirionment. Enter the AD Domain Name as the value for domainName and select the parent OU you want new OUs to be created in  for ou1. You can see in my example, the domain is lab.virtualwin.org and the Parent OU is Lab Machines.

Configure_Workflow
(Click for larger image)

Next, use the Assign a state change workflow to a blueprint and its virtual machines workflow to attach the new workflow to the BuildingMachine stage of a test blueprint. This workflow is located under root > Library > vCloud Automation Center > Infrastructure Administration > Extensibility in the Workflow tree.

To do this, right-click the workflow and select Start Workflow.

Start_Workflow

Choose BuildingMachine as the stub to enable and choose your IaaS host. Remember that we are assuming your IaaS Plugin is already configured. If you don’t see any hosts in this list, you still need to do that! Click Next.

Select_Stub_and_vRA_Host

Now,  select the blueprint(s) you wish to add this workflow to. In this example the selected blueprint is “Add to OU Test” and click Next.

Select_Blueprints

On the last screen, you will be prompted to select the workflow and some final options. Choose the new workflow you just imported (in this example, it is Create OU and Stage Machine). Be sure to choose Yes for Add vCO workflow inputs as blueprint properties and then click Submit.

Add_vRO_Workflow_Inputs

The workflow will complete. Now, switch to your vRealize Automation console and edit the blueprint which you just attached the workflow to. Select the Properties tab. You will be presented with a list of properties, some of which need to be adjusted. The total list of properties you see may vary from environment to environment. Here, we need to delete the following 4 properties:

  • ExternalWFStubs.BuildingMachine.vCACHost
  • ExternalWFStubs.BuildingMachine.vCACVm
  • ExternalWFStubs.BuildingMachine.vCenterVm
  • ExternalWFStubs.BuildingMachine.virtualMachineEntity

And also edit the one named ExternalWFStubs.BuildingMachine.ouName so that Prompt User is checked.

Blueprint_Custom_Properties_Before

When you’re done, the properties should look more like this:

Blueprint_Custom_Properties_After

Now, let’s make that variable a little more friendly. Open the Property Dictionary from the menu on the left. Click on New Property Definition and fill in the data as follows:

  • Name: ExternalWFStubs.BuildingMachine.ouName
  • Display Name: Create New OU to host new VM
  • Control Type: Textbox
  • Required: Yes

Property_Dictionary

That’s it! Now, if you navigate to your vRA Catalog and request the blueprint you’ve been working on, you should see something similar to this.

Request_New_Item

Click Submit and wait for provisioning to complete. When you’re done, you will see the new machine in your Items tab as usual:

Deployed_Items

But if you check out your Active Directory, you should also see that the new OU you selected was created, and the new machine was created inside it!

AD_Properties

Now, this example workflow is a very quick demonstration of concept. It doesn’t have any error handling (and suffice it to say, should NOT be used in any production environments and is provided without support or warranty of any kind) – but it should show you how a seemingly complex  task like this can be accomplished relatively easily. The logic in the workflow could easily be amended to remove the OU creation step. ASD and vRO Dynamic Types could be leveraged to provide the user a list of OUs to choose from, rather than a free-form textbox. The sky’s the limit when it comes to vRA extensibility!

Today’s spicy orchestration experience was brought to you by the Habanero Mojito at Havana, Walnut Creek. Jon_Kate_Havana

I hope this post has been useful.

Creating new vRA Workloads in a specific AD OU

Deploying vRealize Automation Workloads from Apple Watch

Like many people (although not as many as would have liked, I suppose,) I got my shiny new Apple Watch yesterday.

Once it was set up and on my wrist, the first thing I thought of was naturally “How can I use this with vRA?”

Naturally.

It didn’t take long to figure this one out. I don’t know how valuable it will be in the real world, but you have to admit – it sure is cool, particularly for showing the amazing flexibility of vRealize Automation.

Basically, we started with this. A simple button on the Apple Watch that starts a Workflow  which is then handed off to my iPhone.

Apple_Watch_Workflow_Red

Apple_Watch_Workflow_Screenshot

Workflow_Running_on_Apple_Watch
(Edit: I literally just this second learned how to screenshot on the Watch, so I have included both images. Just because)

The iPhone then connects via SSH to a Linux host running CloudClient and runs a deployment script I wrote

Workflow_Details_on_iPhone

The script is quite basic and is as follows:

#!/bin/sh
#
# Autodeploy a vRA item CentOS-vCO Test
echo "---------------------------------------------" >> vRA-Deploy.log
echo "Deployment started at" `date` >> vRA-Deploy.log
/root/cloudclient-3.2.0-2594179/bin/cloudclient.sh vra catalog request submit --id '"CentOS-vCO Test"' --groupid '"Ops Managers"' --reason '"Deployed via Apple Watch"' --export /tmp/request.txt >> vRA-Deploy.log
echo "Deployment handed off to vRA at" `date` >> vRA-Deploy.log
echo "---------------------------------------------" >> vRA-Deploy.log

This uses the auto-login configuration of CloudClient to connect to my vRealize Automation instance and deploy a simple CentOS blueprint from my catalog.

Cloud_Client_Log
(Click image for a larger version)

The details of the deployment come up in the vRA-Deploy.log file…

vRealize_Automation_Apple_Watch_Request_Successful

Deployed_Workload_in_vSphere

And voila! I’ve just provisioned a VM from my watch. The future is now, people.

Edit: 4/26/2015 – I just realized that the step of handing off the workflow from the Watch to the iPhone was unnecessary. The Watch can execute the SSH commands directly without the added handoff. I’ve updated the screenshots and the text accordingly. Cool!

vRealize Operations Manager Content Pack for Log Insight

Get ready, ops-heads… another exciting announcement from the VMware team. There’s now a formal content pack for Log Insight that will allow the import and visualization of the logs from vRealize Operations Manager 6.x.

As an added bonus, if you are running vROps 6.0.1 or later – the Log Insight agent is already pre-installed on your appliance – all you have to do is configure it! If you’re on an earlier version, you can still manually install and configure the agent. Instructions for doing that can be found here.

Given the incredible volume and depth of the data that’s being imported and analyzed by this  content pack, the configuration file is pretty complex. The official installation notes are in a PDF format that was a little difficult to copy and paste all the elements from, so I’ve created a properly formatted file and attached it below.

There are a few tags you will need to change to make this work – I’ve included the tag names as well as the current find-and-replace value below so you can easily tailor the file to your needs. When you’re done, just save it as /var/lib/loginsight-agent/liagent.ini on each node and restart the Log Insight agent (by running /etc/init.d/liagentd restart)

Here’s a helpful screenshot of where you can find several of these parameters for your cluster nodes. Keep in mind that if you have a multi-tier deployment, you will need to customize the below config file for each node.

vRealize_Operations_Manager_Cluster_Administration
(Click the image for a larger version)

Here are the paramters that need to be changed:

  • hostname – this is the IP or FQDN of your Log Insight server. Note that this only needs to be changed in the [server] section at the top of the file, and not throughout the entire file. Below,  it is set to <YOUR LOGINSIGHT HOSTNAME HERE>
  • vmw_vr_ops_clustername – this is the *name* of your vRealize Operations cluster. This can be anything you like here and can be used to distinguish one cluster from another if you have multiples. Below, it is <YOUR CLUSTER NAME HERE>
  • vmw_vr_ops_clusterrole – this is the role that the node you are installing this file on fills. The choices are “Master“, “Replica“, “Data“, or “Remote Collector” – on a single-node installation, use Master. Below, it is set to Master. This value can be found on the Administration > Cluster Management page in the vRealize Operations Manager UI (see above image)
  • vmw_vr_ops_hostname – this is the hostname of your vRealize Operations Manager cluster. This hostname can also be found on the Administration > Cluster Management page in the vRealize Operations Manager UI (see above image). Below, it is set to <YOUR VROPS HOSTNAME HERE>
  • vmw_vr_ops_nodename – this is the node name of the node you are installing this file on. This name can be found on the Administration > Cluster Management page in the vRealize Operations Manager UI (see above image). Below, it is set to <YOUR NODE NAME HERE>

And here’s the config file itself:

; Client-side configuration of VMware Log Insight Agent
; See liagent-effective.ini for the actual configuration used by VMware Log Insight Agent

[server]
; Log Insight server hostname or ip address
; If omitted the default value is LOGINSIGHT
hostname=<YOUR LOGINSIGHT HOSTNAME HERE>

; Set protocol to use:
; cfapi - Log Insight REST API
; syslog - Syslog protocol
; If omitted the default value is cfapi
;
;proto=cfapi

; Log Insight server port to connect to. If omitted the default value is:
; for syslog: 512
; for cfapi without ssl: 9000
; for cfapi with ssl: 9543
;port=9000

;ssl - enable/disable SSL. Applies to cfapi protocol only.
; Possible values are yes or no. If omitted the default value is no.
;ssl=no

; Time in minutes to force reconnection to the server
; If omitted the default value is 30
;reconnect=30

[storage]
;max_disk_buffer - max disk usage limit (data + logs) in MB:
; 100 - 2000 MB, default 200
;max_disk_buffer=200

[logging]
;debug_level - the level of debug messages to enable:
;   0 - no debug messages
;   1 - trace essential debug messages
;   2 - verbose debug messages (will have negative impact on performace)
;debug_level=0

[filelog|messages]
directory=/var/log
include=messages;messages.?

[filelog|syslog]
directory=/var/log
include=syslog;syslog.?

[filelog|ANALYTICS-analytics]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"ANALYTICS","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log
include = analytics*.log*
exclude_fields=hostname

[filelog|COLLECTOR-collector]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"COLLECTOR","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log
include = collector.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|COLLECTOR-collector_wrapper]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"COLLECTOR","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log
include = collector-wrapper.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\.\d{3}

[filelog|COLLECTOR-collector_gc]
directory = /data/vcops/log
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"COLLECTOR","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
include = collector-gc*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\w]\d{2}:\d{2}:\d{2}\.\d{3}

[filelog|WEB-web]
directory = /data/vcops/log
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"WEB","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
include = web*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|GEMFIRE-gemfire]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"GEMFIRE","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log
include = gemfire*.log*
exclude_fields=hostname

[filelog|VIEW_BRIDGE-view_bridge]
tags = {"vmw_vr_ops_appname":"vROps","vmw_vr_ops_logtype":"VIEW_BRIDGE","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log
include = view-bridge*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|VCOPS_BRIDGE-vcops_bridge]
tags = {"vmw_vr_ops_appname":"vROps","vmw_vr_ops_logtype":"VCOPS_BRIDGE","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log
include = vcops-bridge*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|SUITEAPI-api]
directory = /data/vcops/log
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"SUITEAPI","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
include = api.log*;http_api.log*;profiling_api.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|SUITEAPI-suite_api]
directory = /data/vcops/log/suite-api
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"SUITEAPI","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
include = *.log*
exclude_fields=hostname
event_marker=^\d{2}-\w{3}-\d{4}[\s]\d{2}:\d{2}:\d{2}\.\d{3}

[filelog|ADMIN_UI-admin_ui]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"ADMIN_UI","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log/casa
include = *.log*;*_log*
exclude_fields=hostname

[filelog|CALL_STACK-call_stack]
tags = {"vmw_vr_ops_appname":"vROps","vmw_vr_ops_logtype":"CALL_STACK", "vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>","vmw_vr_ops_clusterrole":"Master", "vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>","vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log/callstack
include = analytics*.txt;collector*.txt
exclude_fields=hostname

[filelog|TOMCAT_WEBAPP-tomcat_webapp]
tags = {"vmw_vr_ops_appname":"vROps","vmw_vr_ops_logtype":"TOMCAT_WEBAPP","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log/product-ui
include = *.log*;*_log*
exclude_fields=hostname

[filelog|OTHER-other1]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"OTHER","vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master","vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log
include = aim*.log*;calltracer*.log*;casa.audit*.log*;distributed*.log*;hafailover*.log;his*.log*;installer*.log*;locktrace*.log*;opsapi*.log*;query-service-timer*.log*;queryprofile*.log*;vcopsConfigureRoles*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3} 

[filelog|OTHER-other2]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"OTHER", "vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master", "vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log
include = env-checker.log*
exclude_fields=hostname
event_marker=^\d{2}\D{1}\d{2}\D{1}\d{4}\s\d{2}:\d{2}:\d{2}

[filelog|OTHER-other3]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"OTHER", "vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master", "vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log
include = gfsh*.log*;HTTPPostAdapter*.log*;meta-gemfire*.log*;migration*.log*
exclude_fields=hostname

[filelog|OTHER-watchdog]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"OTHER", "vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master", "vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log/vcops-watchdog
include = vcops-watchdog.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|ADAPTER-vmwareadapter]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"ADAPTER", "vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master", "vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log/adapters/VMwareAdapter
include = *.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|ADAPTER-vcopsadapter]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"ADAPTER", "vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master", "vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log/adapters/VCOpsAdapter
include = *.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|ADAPTER-openapiadapter]
tags = {"vmw_vr_ops_appname":"vROps", "vmw_vr_ops_logtype":"ADAPTER", "vmw_vr_ops_clustername":"<YOUR CLUSTER NAME HERE>", "vmw_vr_ops_clusterrole":"Master", "vmw_vr_ops_nodename":"<YOUR NODE NAME HERE>", "vmw_vr_ops_hostname":"<YOUR VROPS HOSTNAME HERE>"}
directory = /data/vcops/log/adapters/OpenAPIAdapter
include = *.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

See what I mean about complex? And speaking of which… (come on, you had to know this was coming)

Today’s message has been brought to you by Talisman’s 2010 Adastra Vineyard Pinot Noir. The amazing folks at Talisman produce incredible small batch Pinot Noir from several vineyards across northern California. Their philosophy is to focus on the terroir of their fruit, so they produce every wine under precisely the same conditions – from crushing to aging to the oak in the barrels, everything is identical but the fruit itself. This allows the complexities afforded by each individual vineyard to really shine through. This is one of my favorites, with vanilla, dark fruit, spices and a nose that almost makes you forget to take a sip.

Talisman_2010_Adastra

Now. Once you’ve configured and restarted your Log Insight agents on the vRealize Operations Manager cluster nodes, all you have to do is import the Content Pack into Log Insight. It is available for direct download from the VMware Solution Exchange here, or you can install it directly from your Log Insight console by accessing the Content Pack Marketplace and selecting the VMware – vR Ops 6.x Content Pack.

Content_Pack_Marketplace

When that’s  complete, you’re ready to start leveraging the 12 Dashboard Groups, 81 Dashboard Widgets, 18 Queries, 8 Alerts and 31 Extracted Fields that this content pack exposes to you. Check it out!

Log_Insight_vRealize_Operations_Dashboards
(Click the image for a larger version)

It’s also worth noting that if you had previously configured vROps 6.0.x to send its logs to Log Insight directly by editing the logger configuration, you should now undo this configuration. Leaving it in place will result in some logs being sent to Log Insight twice, and may even confuse the content pack.

vRealize_Operations_Edit_Logger_Configuration

Cheers, and happy analyzing!

Extending vRealize Operations Actions with the vRealize Orchestrator Solution and Workflow Package

When vRealize Operations Management 6.0 was released, VMware increased the flexibility afforded to administrators by adding the concepts of symptoms, recommendations and actions to the product. As you might expect, symptoms are thresholds or characteristics that define when a problem may have occurred or additional guidance may be needed. Recommendations are a customizable way to define what that additional guidance might be – and actions allow you to automate and carry out that guidance.

Since then, one of the most frequent questions from my customers has been “When will we be able to use vRealize Orchestrator for these?”

I’m pleased to report that VMware has now enabled that capability via the vRealize Orchestrator Solution and Workflow Package for vRealize Operations. This package is available at the VMware Solution Exchange right now, and the purpose of this post is to guide you through the installation and configuration of it. The package adds many frequently-requested workflows, including:

  • Decommission a Host
  • Place a Host into Maintenance Mode
  • Perform a Power Off or Reboot on a Host
  • Manage VM or VM Group Snapshots
  • Migrate a VM or VM Group
  • Power Off, Power On or Reboot a VM or VM Group
  • Reconfigure a VM or VM Group (CPU and Memory settings)
  • Upgrade the VMware Tools for a VM or VM Group

Clicking the links above will bring you to the Solution Exchange portal where you can read more about and download the package. Click the blue “Try” button to initiate the download.

VSX_Download_vRealize_Orchestrator_Solution_and_Workflows_for_vRealize_Operations

Once you have downloaded and extracted the ZIP file, it’s time to start the installation. The first thing you’ll want to do is ensure that both your vRealize Orchestrator and vRealize Operations Manager are registered to the same vCenter instance. This can be done by comparing the data shown in the two screenshots below.

Validate_vRealize_Operations_vCenter_Connection

Validate _vRealize_Orchestrator_vCenter_Connection

As you can see above, both systems are taking to the same vCenter. We’re ready to begin!

First, you will need to import the Workflow package into your vRealize Orchestrator instance. Start by logging in to the Orchestrator Client.

Log_Into_vRealize_Orchestrator

Ensure that your client view is set to Administer

Switch_to_Administrator_View

Then, click on the Import Package button in the upper left of the right-hand panel.

Import_vRealize_Orchestrator_Package

Select the Remediation Actions Package (default filename is com.vmware.vrops.remediationactionsall-v15.package) and select Open

Select_Package_to_Import

You will be prompted to verify the software signature. Continue by selecting Import

Accept_Package_Signature

vRealize Orchestrator will then present you with a list of all of the new and changed elements that this package import will affect. No changes here are necessary, simply continue by clicking Import Selected Elements

Import_vRealize_Orchestrator_Package_Elements

Once the import completes, you will be able to view the new workflows. Click the Workflows tab to verify that there’s a whole bunch of new vRealize Operations Manager goodness present.

View_Imported_Workflows

You can also verify that the new workflows are present by switching back to the Run view, clicking the Workflows tab and expanding the new vRealize Operations Manager folder. You can see I already have a ton of great workflows by my friends Eric at Cloud Relevant and Sid at Daily Hypervisor in here.

Switch_to_Run_View_and_View_New_Workflows

That’s it for the vRealize Orchestrator side of things. Now you will need to switch over to your vRealize Operations Manager portal. Log in as a user with appropriate rights to add/update solutions. An admin user will work nicely.

Click on the Administration button, followed by the Solutions section. Then, click the Green + to add a new solution.

Import_New_vRealize_Operations_Solution

Select the solution file using the Browse button and click Upload. Once the upload completes and the PAK file has been verified, click Next to proceed with the installation.

Select_Solution_PAK

Accept the EULA and click Next again. Wait for the installation to complete, then select Finish

Complete_Solution_Installation

You can now verify that your new solution is installed by locating the vRealize Orchestrator Actions Adapter in the solutions list. Note that you may have to scroll down to find it, if you have several solutions installed. You may also notice that the adapter instance is not yet configured. Let’s tackle that next!

Verify_New_Solution_is_Installed

To configure the adapter instance, ensure that the vRealize Orchestrator Actions Adapter is still selected, then click the Gears icon at the top, next to the Green + we clicked a few steps back.

Give your new adapter a name, and enter the IP or hostname of your vRealize Orchestrator instance. Be sure to use the same Orchestrator instance as we verified at the beginning of this process. Click the Green + to add credentials for the instance.

Configure_New_vRealize_Operations_Solution

Enter your credentials and click OK

Add_New_Credential

Next, click Test Connection. You may be presented with a certificate warning – click OK if you trust the certificate, and then your test should be successful!

Accept_vRealize_Orchestrator_Certificate

Solution_Test_Successful

Save your new adapter by clicking Save Settings and finally Close the configuration dialog.

That’s it for the installation! You can verify that the new actions are present by clicking on the Content tab inside vRealize Operations and selecting Actions from the list on the left. If all went well, you should see the 8 new actions present. These can now be combined with symptoms and recommendations to unlock many new possibilities for remediation inside your environment.

View_New_Available_Actions
(Click for larger image)

Since it’s not even 9am yet, today’s post will be brought to you by the Zesty Bacon Bloody Mary from the Boon Fly Cafe in Napa, CA. This exceptional libation combines top-shelf Vodka with Boon Fly’s own special spice blend, a celery salt rim and a massive slab of applewood smoked bacon to top it all off. Paired with Boon Fly’s fresh made donuts, it’s the best breakfast in the valley. Bloody Marys also have the (dubious?) honor of being the drink that’s OK to have first thing in the morning. After all, you’re not an alcoholic, you’re just a little tired.

Bacon_Bloody_Mary_Boon_Fly_Cafe

I hope this guide has proved useful and that you have a chance to head out to Boon Fly and try their delicious concoctions.