This was a challenge I set myself, but this challenge does not apply to Lets Encrypt certificate requests (or any other external certificate service) or certificate renewals that is handled beautifully by Certbot.
This entry is based on renewing certificates which are hosted on non-Windows operating systems and where the certificate authorities are windows platforms - while this can introduce certain challenges for example PKCS8 v PKCS1 - that can quite easily be overcome with some OpenSSL commands.
The prerequisite requires ADCS (Active Directory Certificate Server) and you need to have the web-service management enrollment feature enabled, this is the website usually on the address http://<certsrv>/certsrv this will allow you to request and download the response straight from the server.
The private key will remain on the server requesting the CSR and the scripts will take care of the certificate generation on the CA, so lets get to it, I find its best to have an action plan with this, and this is how I got it to work, feel free to customise this as you need, or not.
Step 1 : Generate the CSR
Step 2 : Request the new certificate from the CA with a file with the request ID
Step 3 : Cleanup the file that includes the request ID
Step 4 : Request the download from the CA (manually updating the Request ID)
Step 5 : Request the download from the CA (using the ReqID.txt file)
Step 6 : Promote the new certificate to the live website and restart Apache2
This is the process as above, this is a slow and steady approach, but this does work as it should.
Making scripts Executable
Once you have saved your scripts you need to make then executable to work, this needs to be completed to run them, you can do that as you create them or at the end.
sudo chmod +x <script.sh>
You need to ensure that from the directory all your executable scripts are green in colour which means they can be executed - if not you will get "Permission Denied"
Request CSRThis is the first step and for this you need a configuration file for this, in this example we will call it openssl-san.cnf so to create the file we can use this, once the command below in in the terminal press enter, job done....
cat > sslcert-san.cnf <<EOL
[ req ]
default_bits = 2048
default_keyfile = mykey.key
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no
[ req_distinguished_name ]
C = UK
ST = Bear County
L = Bearville
O = Acme Bears
OU = Honey Extractors
CN = demo.pokebearswithsticks.com
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = demo.pokebearswithsticks.com
EOL
You have now created the openssl-san.cnf file for the next section, this is where we create the CSR and the private keys, you can enter this into the shell but I have called this:
Script 1 : Named - 1_generate_csr.sh
openssl req -new -config openssl-san.cnf -keyout mykey.key -out myrequest.csr
Script 2 : Named - 2_request_withfile.sh
You will need to update the variables in bold and then when the script is complete you will have a file called request_id.txt in the current folder you are in, this will contain 4 x Request ID numbers.
# Define variables
CA_SERVER="http://bearca.bear.local/certsrv"
USERNAME="bear\\ca-request"
PASSWORD="<SecurePassword>"
CSR_FILE="myrequest.csr"
DER_CSR_FILE="myrequest.der"
CERT_FILE="mycertificate.cer"
REQUEST_ID_FILE="request_id.txt"
CERT_TEMPLATE="WebServer"
COOKIES_FILE=$(mktemp)
# Convert the CSR to DER format
openssl req -in $CSR_FILE -outform DER -out $DER_CSR_FILE
# Step 1: Get initial session cookies
curl -s -S --noproxy '*' --ntlm --user $USERNAME:$PASSWORD \
-c $COOKIES_FILE \
$CA_SERVER > /dev/null
# Step 2: Submit the CSR with the correct form data
RESPONSE=$(curl -s -S --noproxy '*' --ntlm --user $USERNAME:$PASSWORD \
-b $COOKIES_FILE -c $COOKIES_FILE \
-X POST \
-H "Content-Type: application/x-www-form-urlencoded" \
--data-urlencode "Mode=newreq" \
--data-urlencode "CertRequest=$(base64 -w 0 < $DER_CSR_FILE)" \
--data-urlencode "CertAttrib=CertificateTemplate:$CERT_TEMPLATE" \
--data-urlencode "FriendlyType=Saved-Request" \
--data-urlencode "TargetStoreFlags=0" \
--data-urlencode "SaveCert=yes" \
$CA_SERVER/certfnsh.asp)
# Extract the request ID from the response
REQUEST_ID=$(echo "$RESPONSE" | grep -oP 'ReqID=\K[0-9]+')
# If the request ID is not found, log an error and exit
if [ -z "$REQUEST_ID" ]; then
echo "Error: Request ID not found in the response."
echo "Response from the CA server:"
echo "$RESPONSE"
exit 1
fi
# Log the extracted request ID
echo "Extracted Request ID: $REQUEST_ID"
# Write the request ID to a text file
echo "$REQUEST_ID" > "$REQUEST_ID_FILE"
echo "Request ID written to: $REQUEST_ID_FILE"
# Clean up temporary files
rm $COOKIES_FILE
The reason you get 4 x Request ID numbers, and the reason for the 4 x numbers is you have got to this point in the process, which means you have a certificate but you get 4 x download options, there are 2 x DER certificate and 2 x Base64 certificate - we only care about the Base64 certificate, but for this example we only require the Request ID (the next script will take care of the Base64 certificate process)
Script 3 : Named - 3_cleanup_reqid.sh
This is the script that needs to be run in Python and not Bash, so for this I have used Pyhton3 and this is the contents of this file, the job is this file is simple, that is to take the output of 4 x Request ID's and then make it a single ID.
# Python script to read request_id.txt, extract a single unique request ID, and write it to ReqID.txt
# Specify the path to your request_id.txt file
input_file_path = 'request_id.txt'
# Specify the path to the output file
output_file_path = 'ReqID.txt'
# Read the file and extract unique request ID
with open(input_file_path, 'r') as input_file:
lines = input_file.readlines()
# Extract the first line (assuming all lines are the same and contain the same request ID)
if lines:
unique_request_id = lines[0].strip()
print(f'Unique request ID extracted: {unique_request_id}')
# Write the unique request ID to ReqID.txt
with open(output_file_path, 'w') as output_file:
output_file.write(unique_request_id + '\n')
print(f'Unique request ID written to {output_file_path}')
else:
print('No data found in the input file.')
You will need to run this as:
python3 3_cleanup_reqid.sh
Once this script is executed you will end up with a file called ReqID.txt which should contain a single request ID - and that request ID is for the certificate that you need to "request" from the CA server.
Script 4 : Named - 4_request_download.sh
This is the "proving the process works" file and in this file you need to manually update the Reest ID with the number in the ReqID.txt file.
Why the manual update?
Interesting question, the reason is was like this was due to an error when I tried to script it into one file, it would appear "curl" has some issues with variables in this circumstance, this took me quite a while to figure out and after lots of HTTP 401 or Unauthorised errors and the fact the command would not parse
This was driving me nuts as with the complete script this was the error, due to many issues with the syntax not working:
curl: (3) URL using bad/illegal format or missing URL
Content of the certificate file (Base64 format):
cat: mycertificate.cer: No such file or directory
Can't open mycertificate.cer for reading, No such file or directory
I tried to curl the command I required with this and this returned a HTTP 200 and the certificate from the CA, excellent, so this was a test of that variable being set manually in the script.
curl -v --noproxy '*' --ntlm --user bear.local\\ca-request:magicalme "http:/bearca.bear.local/certsrv/certnew.cer?ReqID=2888161"
When that above command was run rather than the error we got the certificate as below:
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html
< Server: Microsoft-IIS/15.0
< Set-Cookie: ASPSESSIONIDCCACTTAB=PDILBDDAECLFGAKADNNNIFBJ; path=/
< Persistent-Auth: true
< Date: Mon, 12 Jun 2024 16:37:09 GMT
< Content-Length: 2358
<
-----BEGIN CERTIFICATE-----
MIIGhDCCBWygAwIBAgITIQAcz6EJv3PgCaiFDgAAABzPoTANBgkqhkiG9w0BAQsF
ADBKMRUwEwYKCZImiZPyLGQBGRYFaW50cmExFzAVBgoJkiaJk/IsZAEZFgdzdHdh
dGVyMRgwFgYDVQQDEw9TVFctSXNzdWluZy1DQTEwHhcNMjQwNjE3MDg1MjI4WhcN
MjYwNjE3MDg1MjI4WjBxMQswCQYDVQQGEwJVUzEOMAwGA1UECBMFU3RhdGUxDTAL
BgNVBAcTBENpdHkxFTATBgNVBAoTDE9yZ2FuaXphdGlvbjETMBEGA1UECxMKRGVw
YXJ0bWVudDEXMBUGA1UEAxMOeW91cmRvbWFpbi5jb20wggEiMA0GCSqGSIb3DQEB
Now we have the correct syntax for this download, we need to confirm this works in a script with the manual ReqID variable being manually updated as below:
# Define variables
CA_SERVER="http://bearca.bear.local/certsrv"
USERNAME="bear\\ca-request"
PASSWORD="<SecurePassword>"
REQUEST_ID="1888161" # Replace with your actual request ID
CERT_FILE="mycertificate.cer"
# Construct the download URL for the Base64 encoded certificate
CERT_URL="$CA_SERVER/certnew.cer?ReqID=$REQUEST_ID&Enc=b64"
# Log the constructed URL (optional)
echo "Certificate URL: $CERT_URL"
# Download the issued certificate in Base64 encoded format using curl
curl -v --noproxy '*' --ntlm --user "$USERNAME:$PASSWORD" \
-o "$CERT_FILE" \
"$CERT_URL"
# Check if the certificate file was downloaded successfully
if [ $? -ne 0 ]; then
echo "Error: curl command failed to download the certificate."
exit 1
fi
# Verify the content of the certificate file
echo "Content of the certificate file (Base64 format):"
cat "$CERT_FILE"
Script 5 : Named - 5_useReqID_download.sh
This script removes the need to manually update the ReqID with the actual number as it will read it from the file we created from Script 3, the rest of the syntax is essential the same as before, with some slight modifications:
# Define variables
CA_SERVER="http://bearca.bear.local/certsrv"
USERNAME="bear\\ca-request"
PASSWORD="<SecurePassword>"
REQID_FILE="ReqID.txt" # File containing the request ID
CERT_FILE="mycertificate.cer"
# Read the request ID from ReqID.txt
REQUEST_ID=$(cat "$REQID_FILE")
# Construct the download URL for the Base64 encoded certificate
CERT_URL="$CA_SERVER/certnew.cer?ReqID=$REQUEST_ID&Enc=b64"
# Log the constructed URL (optional)
echo "Certificate URL: $CERT_URL"
# Download the issued certificate in Base64 encoded format using curl
curl -v --noproxy '*' --ntlm --user "$USERNAME:$PASSWORD" \
-o "$CERT_FILE" \
"$CERT_URL"
# Check if the certificate file was downloaded successfully
if [ $? -ne 0 ]; then
echo "Error: curl command failed to download the certificate."
exit 1
fi
# Verify the content of the certificate file
echo "Content of the certificate file (Base64 format):"
cat "$CERT_FILE"
Script 6 : Named - 6_promote_live.sh
This is the final script, and this will take the files created by the the previous scripts, create a directory called "ssl-live" and then move those certificates to that folder with the correct name as per the apache configuration file (this is the one in /etc/apache2/sites-available) then you will be asked if you want to move the files to the "live directory" then after you will be asked if you wish to restart Apache2.
# Create the ssl-live directory if it doesn't exist
mkdir -p ssl-live
# Copy and rename the files
cp mycertificate.cer ssl-live/certificate.crt
cp mykey.key ssl-live/certkey.key
# Prompt the user to override the live files
read -p "Do you want to override the live files? (yes/no): " override
if [ "$override" == "yes" ]; then
# Copy files to /etc/ssl/certs/
sudo cp ssl-live/certificate.cer /etc/ssl/certs/certificate.cer
sudo cp ssl-live/keyfile.key /etc/ssl/certs/keyfile.key
echo "Files have been copied to /etc/ssl/certs/"
else
echo "Live files were not overridden."
fi
# Prompt the user to restart apache2
read -p "Do you want to restart apache2? (yes/no): " restart
if [ "$restart" == "yes" ]; then
# Restart apache2
sudo systemctl restart apache2
echo "apache2 has been restarted."
else
echo "apache2 was not restarted."
fi
If you have said "yes" to moving the files and restarting Apache2 then your new certificate should now be live, if you wish this to be an automated task then you can remove the prompt for "do you wish to to do x" then you can run the scripts one after the other if you like.
Script : Executable Permissions
You now need to ensure all your scripts are executable, if you have used the name files names as me you can run these commands:
chmod +x 1_generate_csr.sh
chmod +x 2_request_withfile.sh
chmod +x 3_cleanup_reqid.sh
chmod +x 4_request_download.sh
chmod +x 5_useReqID_download.sh
chmod +x 6_promote_live.sh
Script : Named - AutoRenew.sh
This will run all the scripts one by one and then you will end up with this process being automatic (excluding the confirmations in the last script)
# Generate CSR and Key
./1_generate_csr.sh
# Submit CSR with RefID file
2_request_withfile.sh
# Cleanup RefID file
python3 3_cleanup_reqid.sh
# Download CER file from RefID
5_useReqID_download.sh
#Enable new certificate and restart Apache2
6_promote_live.sh
