Tag: Automation

  • Homebrew Installed Jenkins on MacOS Sequoia Cannot be Accessed From External

    I recently installed Jenkins 2.479.2 on MacOS Sequoia (15.1). The service is working fine on http://localhost:8080, but it can not be accessed from any other devices at my home.

    This post provided solutions to fix that issue. Here is a summary:

    1. Make sure the MacOS firewall is disabled, or the port is enabled on the firewall.
    2. Replace 127.0.0.1 with 0.0.0.0 in the file /opt/homebrew/opt/jenkins-lts/homebrew.mxcl.jenkins-lts.plist.
    3. Restart the Jenkins service with brew services restart jenkins-lts.
      • You can use the command to restart other brew installed services. Use brew services info --all to list all available services.

  • Fixing Motor Unresponsiveness on Raspberry Pi Due to Grounding

    The last time I wrote about Raspberry Pi was Connect to New Provisioned Raspberry Pi Less than $3, about six years ago. Time flies! Today, I will talk about a grounding issue.

    Background and Troubleshooting

    I recently purchased a powerful motor driver board. It supports two DC motors with encoders and other features. During the initial testing, I experienced weird behavior. As a cloud engineer, I find it interesting to learn about the physical world.

    (more…)
  • Fixing ‘KDC Reply Did Not Match Expectations’ Error in Ansible for Windows Server

    When you connect to a Windows server with Ansible, you may see the error message “KDC reply did not match expectations while getting initial credentials”.

    One possible reason is that the domain name that you used in the Ansible playbook command line doesn’t match the domain name that is configured in the Linux krb5.conf.

    The domain name is case-sensitive.

  • Terraform Cloud – How to List All Users

    Terraform Cloud – How to List All Users

    Terraform Cloud has a rich API. However, the API documentation does not mention how to list all users. We can leverage the organization membership API and the PowerShell command Invoke-RestMethod to get a user list.

    (more…)
  • Connect-NsxtServer shows “Unable to connect to the remote server”

    When you run Connect-NsxtServer in the PowerCLI, it may show “Unable to connect to the remote server“.

    Because the error message is a little bit confusing with other login issues. It’s not easy to troubleshoot. The actual reason is the NSX-T uses a self-signed certificate, and the PowerCLI cannot accept the certificate automatically.

    The fix is super easy. You need to set the PowerCLI to ignore the invalid certificate with the following command:

    Set-PowerCLIConfiguration -Scope User -InvalidCertificateAction:Ignore -Confirm:$false
  • Move Terraform Providers to Other Folders

    Create a new control file with the name .terraformrc or terraform.rc in your profile folder.

    Add the following lines:

    plugin_cache_dir   = "$HOME/.terraform.d/plugin-cache"

    Create the folder .terraform.d/plugin-cache in your profile folder.

    The providers will be downloaded to the cache folder when you run terraform init.


    If you don’t want to create the control file in the profile folder. Alternative is to create an environment variable.

    export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"
    
  • Setup Terraform and Ansible for Windows provisionon CentOS

    Setup Terraform and Ansible for Windows provisionon CentOS

    Provisioning Windows machines with Terraform is easy. Configuring Windows machines with Ansible is also not complex. However, it’s a little bit challenging to combine them. The following steps are some ideas about handling a Windows machine from provisioning to post configuration without modifying the winrm configuration on the guest operating system.

    1. Install required repos for yum.
    yum -y install https://repo.ius.io/ius-release-el7.rpm
    yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
    yum -y install https://packages.endpointdev.com/rhel/7/os/x86_64/endpoint-repo.x86_64.rpm
    yum -y install epel-release
    yum -y install yum-utils
    yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
    ShellScript
    1. Install Terraform.
    sudo yum -y install terraform
    ShellScript
    1. Install Ansible.
    sudo yum -y install ansible
    ShellScript
    1. Install Kerberos.
    yum -y install gcc python-devel krb5-devel krb5-libs krb5-workstation
    ShellScript
    1. Install pip.
    sudo yum -y install python-pip
    
    # You probably need the following packages if you are using VPN
    pip install pysocks
    ShellScript
    1. Install pywinrm[kerberos].
    pip install pywinrm[kerberos]
    ShellScript
    1. Configure /etc/krb5.conf.
      The following are the required lines. Please make sure to change the domain name to yours. And it’s case-sensitive.
    [libdefaults]
     dns_lookup_realm = true
     dns_lookup_kdc = true
     forward = true
     forwardable = true
     default_realm = ZHENGWU.ORG
    
    
    [realms]
     ZHENGWU.ORG = {
      kdc = DC.ZHENGWU.ORG
      admin_server = DC.ZHENGWU.ORG
     }
    
    [domain_realm]
     .zhengwu.org = ZHENGWU.ORG
     zhengwu.org = ZHENGWU.ORG
    ShellScript
    1. Create an Ansible inventory file.
    [win] #Group name
    dc.zhengwu.org #This is the target server list
     
    
    [win:vars]
    ansible_connection=winrm 
    
    ansible_user=administrator #It's better a domain admin account.
    ansible_password=P@ssw0rd #Change this password
    ansible_port=5985
    ansible_winrm_transport=kerberos
    ansible_winrm_server_cert_validation=ignore
    ShellScript
    1. Run Ansible win_ping test.
    ansible <group in inventory file> -m win_ping -i <inventory file>
    ShellScript
  • Quick Note: Modify PowerShell Execution Policy in Registry

    I think to maintain a solid security policy is mandatory in an enterprise network. The information leak not only a threat to IT but also impact the firm’s profits and stock performance. Firms usually leverage Active Directory Group Policy to control security settings on Windows. The PowerShell execution policy is the one most company ITs would restrict.

    IT Pros may need to do some testing in scripting. I have posted how to change PowerShell execution policy by PowerShell command.

    Following is the step of how to get rid of the restriction. This is just for temporary use. You should revert it immediately after testing!!!

    1. Open Registry Editor.
    regedit.msc
    1. Go to HKLMSOFTWAREMicrosoftPowerShell1ShellIdsMicrosoft.PowerShell.
    2. Change the registry key ExecutionPolicy value to RemoteSigned.
    3. Open a new PowerShell prompt and run your script.

    If you don’t want to change the registry key. You can paste the scripts into Windows PowerShell ISE and run all of the lines. Please note this option may throw error if you have relative paths in the script due to the temp script running under the same directory of Windows PowerShell ISE application.

    How to run selection in Windows PowerShell ISE
    How to run selection in Windows PowerShell ISE

    Please refer to Microsoft document if you want to learn more about PowerShell execution policy.

  • How to Manage Windows Servers With Ansible on CentOS 8

    Ansible is a popular automation tool for infrastructure configuration. It runs on the Linux system. CentOS is an ideal distribution to run Ansible for lab purposes. It is similar to the Red Hat Linux but free. And the latest major release is CentOS 8. It contains Python 3 by default. So the Ansible configuration is different from CentOS 7. I will focus on the configuration in the lab environment. The goal is to create a simple environment to manage Windows servers with Ansible.

    Ansible Installation on CentOS 8

    I used CentOS 8 mini installation. It has no extra software installed. The procedure below maybe a bit different from your environment if you installed other roles on the OS.

    Ansible is a standalone application that not rely on databases. There are two files it mainly needed in a quick lab environment: Playbook and host files. You can install multiple Ansible servers. They can run independently to control the same group of Windows servers.

    I would suggest you take a snapshot before moving forward if your Ansible will running on a virtual machine.

    1. Enable Extra Packages for Enterprise Linux for yum.
    yum install epel-release
    1. Install Ansible
    yum install ansible
    1. (Optional) Install pip for Python 3. This step is for Red Hat 8.
    yum install python3-pip
    1. Install pywinrm. The pywinrm will be used to communicate to Windows servers via winrm.
    pip3 install pywinrm
    1. Install dependencies for pywinrm to use Kerberos in order to authenticate to Active Directory.
    yum install gcc python3-devel krb5-devel krb5-libs krb5-workstation
    pip3 install pywinrm[kerberos]

    Ansible installation is completed. The procedure is elementary level but I spent some time figuring it out. Especially the Kerberos and pywinrm parts. 🙂

    Please go to pywinrm GitHub if you want to dig into it.

    Ansible Configuration on CentOS 8

    Configure Ansible

    As I mentioned in the previous section. There are two main files: Playbook and host. A Playbook is a file consist of multiple tasks that will run on the target Windows servers. It’s not covered by this article. The host file stores variables, and target server FQDNs or IP addresses. Ansible gets the target servers’ information in the host file when you run a playbook.

    The host file location is /etc/ansible/hosts. There are two sections in the file for lab purpose.

    1. Server group. You can have multiple groups. Group name is in [ ]. You can give FQDN or IP addresses of the target Windows servers. I recommend using FQDNs if your targets are domain member servers. My example uses server win2019test1.zhengwu.org.
    [windows]
    win2019test1.zhengwu.org
    1. Variables of the target server group. Since this is for lab purpose. I’ll just list required variables in the /etc/ansible/hosts file. You need to use standalone variable files and avoid to input password if it’s for production. Following is a sample of the variable set for the windows group.
      • Variables are linked to a group by the variable name in the first line: [group name:vars].
      • The domain name should be uppercase in ansible_user. The reason is krb5 requires the uppercase domain name in the configuration file. We should match the name here. The domain name is not required if you use a local account.
      • ansible_winrm_server_cert_validation is optional. It only useful when ansible_winrm_scheme is ‘https‘.
      • ansible_port is ‘5985‘ when ansible_winrm_scheme is ‘http‘. Or ‘5986‘ when ansible_winrm_scheme is ‘https‘.
      • ansible_winrm_transport is ‘kerberos‘ in this example since the target Windows servers are domain members. It can be ‘ntlm‘ if you want to authenticate by local account. There are 5 authentication methods on Windows. Kerberos and NTLM are enabled by default. Please refer to Windows Remote Management for detail.
    [windows:vars]
    ansible_user='[email protected]'
    ansible_password='123321'
    ansible_connection='winrm'
    ansible_winrm_scheme='http'
    ansible_port='5985'
    ansible_winrm_transport='kerberos'
    ansible_winrm_server_cert_validation='ignore'

    Configure Kerberos

    Apart from Ansible configuration. We should configure Kerberos for domain authentication if the target Windows servers are joined a domain. My lab servers are joined domain ‘zhengwu.org‘. We have installed Kerberos components in the Ansible Installation on CentOS 8 section. So we just need to configure it. Edit Kerberos configuration file: /etc/krb5.conf.

    1. Change the default domain name. Make sure to remove # to uncomment the line. The domain name should be uppercase.
    default_realm = ZHENGWU.ORG
    1. Uncomment all lines in the realms section. Please note domain name should be uppercase. The parameters kdc and admin_server are the same for the lab environment. The following is an example:

    [realms]
    ZHENGWU.ORG = {
         kdc = DC.ZHENGWU.ORG
         admin_server = DC.ZHENGWU.ORG
     }
    

    Please refer to MIT Kerberos Documentation for the explanation.

    Now Kerberos is configured. We have configured domain credentials in Ansible configuration file, specified Kerberos as the authentication method, and configured Kerberos for Active Directory. We just need to run the Windows ping module in Ansible to test the connection to target Windows servers. You should complete section Manage Windows servers with Ansible if the testing is failed.

    ansible windows -m win_ping

    You should see following output if authentication is successfully.

    win2019test1.zhengwu.org | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }

    Kerberos troubleshooting

    You may see authentication problem when validating target Windows server connection by Ansible win_ping module. Here is simple steps to troubleshooting Kerberos authentication

    1. Try authenticate to target Windows servers by domain account on Ansible server. It can be any domain account.
    kinit [email protected]
    1. List cached authentication data. You should see something similar below.
    Ticket cache: KCM:0
    Default principal: [email protected]
    Valid starting       Expires              Service principal
    06/26/2020 03:56:12  06/26/2020 13:56:12  krbtgt/[email protected]
            renew until 07/03/2020 03:56:09

    Manage Windows servers with Ansible

    The target Windows servers should be configured to accept the winrm connection. Ansible provides a PowerShell script to configure target Windows servers automatically. The script should not be used in a production environment according to Ansible stated in their document.

    The configuration is super easy for production. Open a command prompt under the administrator permission and then run following command

    winrm quickconfig

    Conclusion

    Manage Windows servers with Ansible is not so hard as long as the authentication is configured correctly. Ansible is not the only tool for automation. I’m a super fan of PowerShell. I have posted some articles for automation you may want to check. PowerShell and Ansible both are automation tools.

    I think manage Windows server with Ansible is like outsourcing PowerShell scripting works to communities. You give inputs to the tasks then Ansible modules will execute pre-defined PowerShell scripts and feedbacks output. Ansible reduces the development time of Windows automation but it still has some disadvantages. Such as you have to run multiple tasks to enable Remote Desktop on target Windows servers which is just a single task in PowerShell DSC. So I think automation of infrastructure is a combination of tools like Swiss Army Muti-Tools, each one has an advantage. We have to use them together to achieve the final goal of automation.

  • [Quick Note] Failed to install pywinrm on CentOS 8

    You may see error message “Running setup.py install for pykerberos … error” when install pywinrm on CentOS 8. The additional errors are “unable to execute 'gcc': No such file or directory” and “command 'gcc' failed with exit status 1“.

    The reason because gcc dependence is missing on the machine. You need to run following command to install gcc then try install pywinrm again.

    # yum install gcc