Tuesday, December 23, 2014

[3 of many] Migrating to Fortinet 5.2 - ECMP Load Balancing - Answers

In the last post here, I have discussed the problems we had with ECMP. In short, the traffic was not balancing properly and switching from one to the other connection after we have migrated to 5.2.

First, the answers from technical support:
A very busy but knowledgeable and fast person has taken our case. After initial testing based on our suggestions, it took an hour or so to rapidly check everything on our system and answer one or two of  my questions.
It looks like everything was working since the beginning. In fact, the documentation from Fortinet

states that ECMP load balancing is using upload (yes you heard it right!- upload) traffic to determine when load balancing occurs. That is, under lab conditions a few Youtube videos saturating 5Mbit connection generate around 600kbits of upload traffic. At the same time, a single Skype conversation will result in around 1Mbit of upload bandwidth used. Given that our own traffic is mostly Youtube (90% to 95%) we have set up spillover threshold to 600k. I will post here any adjustments we make.

A few important notes:
  • ECMP is using the first available route as default if all routes have the same distance. That is, for ECMP to work in proper and predictable manner (according to Fortinet support) all routes must have the same distance. 
  • However, ECMP accepts routes with different distances and is supposed to select the first available route with the shortest distance as the default and the other one as the spillover. This method was perfectly functional just before 5.2 and should also be working after 5.2. It is not an official statement and should be tested.
In short, I was wrong and Fortinet has helped me to find the answers. Unfortunately, the support person answered only the questions we had on the support ticket. In fact, he has really politely suggested to open a new ticket for debugging the new load balancing method (Wan Link Load Balancing) because he is busy and another client is waiting for him. I cannot say that he was unhelpful or impolite but I do expect to have more than an hour of support in the rare occasions I need help and I finally get it from someone who knows what he/she is doing.

Thursday, October 30, 2014

[2 of many] Migrating to Fortinet 5.2 - ECMP Load Balancing

While I have not done hundreds of ours of testing, I'm fairly certain that ECMP Load Balancing method that worked before 5.2 is now partially buggy and does not perform as expected.

We are using the following config:
  • 300C unit
  • 2 WAN connections
  • Spillover load balancing
Fortinet suggests here to do the following:
  • Configure static routes
  • Configure spillover thresholds
  • Configure interface status detection
Static routes
  1. Notice that the distance is set to the same value: in this config, the unit is supposed to select the shortest distance automatically and use it threshold is reached. Well it does not work as we will see in the images bellow.
  2. In the initial setup under FortiOS 5.0, we had ISP0 distance set to 11 so that, according to the latest documentation, all connections go to port9 until threshold is reached. It did work before  we have migrated to 5.2 but is clearly not working now.
 Spillover thresholds and interface status detection

The behavior in FortiOS 5.2

Normal behavior with 30+ users for the past hour
Notice how the second WAN connection is not getting used at all? Considering that there are multiple users and the link gets saturated well above the threshold of 4500kbits set in the ECMP balancing (it gets up to 5.2Mbit=5320kbits), it is a weird behavior that should not occur in a normal usage scenario.

Simulating WAN connection down
However, it looks like fail-over is working???
Will it then load balance after we bring back the main connection?

Well, it does go back to main connection.and completely drops the second one. Despite the fact that during downtime of WAN1 the routes in cash were using WAN2, the system almost immediately comes back to the same old behavior we have noticed earlier: all connections are reset to WAN 1.

Conclusion: spillover does not work. We can at best hope for fail-over.

We can even go farther and diagnose connection behavior:
Let us change the spill-over threshold to 1 for port9.  In CLI, we will go to a VLAN that has the above setup (if any) and type the following command :
diagnose netlink dstmac list
The output is the following:
dev=port9 mac=00:00:00:00:00:00 rx_tcp_mss=0 tx_tcp_mss=0 overspill-threshold=128 bytes=308 over_bps=1 sampler_rate=0
By comparing overspill-threshold (in bytes) and bytes (actual usage in bytes) value we can see that the connection has reached over its new threshold. Moreover, over_bps=1 indicates that the unit has detected the limit and is supposed to forward new connections to the second port. By going to VDOM-->Log and repport-->Traffic Log --> Forward traffic we can examine the behavior and we notice that the spill-over actually works! yes it does! But what has happened previously?
Well, if we put the values back as they were and we generate lots of various traffic from various sources (plus there are some unsuspecting users using the network right now), we get the following:

dev=port9 mac=00:00:00:00:00:00 rx_tcp_mss=0 tx_tcp_mss=0 overspill-threshold=576000 bytes=132 over_bps=0 sampler_rate=0
dev=port9 mac=00:00:00:00:00:00 rx_tcp_mss=0 tx_tcp_mss=0 overspill-threshold=576000 bytes=54 over_bps=0 sampler_rate=0
dev=port9 mac=00:00:00:00:00:00 rx_tcp_mss=0 tx_tcp_mss=0 overspill-threshold=576000 bytes=162 over_bps=0 sampler_rate=0
dev=port9 mac=00:00:00:00:00:00 rx_tcp_mss=0 tx_tcp_mss=0 overspill-threshold=576000 bytes=66 over_bps=0 sampler_rate=0

While the connection looks like this:

Despite the fact that the WAN1-port9 interface is saturated well above spill-over limit, a short inspection of logs shows that no spill-over occurs and all connections that have been previously forced to a second WAN are now back to WAN1.  All this is due to the fact that something is wrong with the setup and/or detection of the traffic: it simply cannot vary between 54 and 162 bytes when we see 5.2Mbit (more than 681 000 bytes) of traffic. Clearly 15 minutes above are not enough to be able to see any effect of load-balancing, especially under lab conditions, but the unit still should indicated that a limit spill-over has been reached (over_bps=1 should be set for port9). 

Unfortunately, I do not have time or energy to investigate this farther. Tomorrow, I and my companion will redo the entire setup and use the new load balancing method. The idea comes from the official Fortinet YouTube channel. Note however that the settings are actually elsewhere in our v5.2.1,build618 (GA) FortiOS: 
VDOM_NAME-->Network--> WAN Link Load Balancing Interface 
or if you do not have VDOMs
System-->Network--> WAN Link Load Balancing Interface 

I do not want to bother fixing the above not because I like so much re-configuring everything but because the new setup has a promise to simplify IPv4 tables and reduce by half the amount of policies we have currently: WAN1 and WAN2 have the same policies. Hopefully it will work as expected

UPDATE 21/10/2014: So we have tried for almost 7 hours to make it work and we failed. We had to revert back to the above described method because the system was unstable: pings and connections were dropping for no known (to us) reason. I have created a ticket with Fortinet and will keep you posted. 

UPDATE 23/12/2014: The answer was easy... but the issue of proper loadbalancing was not solved. See my post:  [3 of many] Migrating to Fortinet 5.2 - ECMP Load Balancing - Answers



Monday, October 27, 2014

[1 of many] Migrating to Fortinet 5.2 - Overview

This is a first of possibly many small remarks on migration process from Fortinet 5.0 to a 5.2 version.

The migration process went on smoothly. In fact, the entire prep and upgrade took barely 15 minutes! Fortinet has multiple advisories warning of all things that will go wrong basically implying that the entire setup may go crazy. In our case, we have seen some duplication of rules and weird behavior but the unit is fully functional and stable enough for a radical change like this one.

For instance, we have seen most web filtering and ssh rules duplicated in a format one rule per user group/type.

BEFORE
AFTER
Similarly, some SSL rules have been duplicated but nothing that cannot be cleaned up in an hour or so.

Unfortunately, some quirks are annoying.

  1. It looks like the old method we have used for load balancing two WAN connections does not work as expected anymore. The spillover does not perform as expected: the unit functions as a fail-over from WAN1 to WAN2. See my next post for more details.
  2. The unit routinely goes from less than 10% load to 100% load. This is unusual for a machine that normally does not even break a sweat and was specifically purchased to exceed possible maximum workloads ensuring multiple years of continuous service.
  3. It is possible that both issues are related. Since the rules are managed and processed in a different manner, there could be a visible advantage (for CPU) in reducing the number of IP rules by levering a new method for WAN load balancing and aggregation.

Reminder of the setup:
  • Fortinet 300C
  • Two WAN connections set up in spillover format 
  • Multiple VLANs on the network (guests, administration, employees, students etc.) 
    • some completely isolated with DHCP managed by Fortigate such as guests
    • some are allowed limited communication between them
  • Fortigate is setup with two VDOMs with limited and controlled connectivity between them
  • Overall, we are talking about something like 100 IPv4 rules with specific web filtering, application control, IPS, SSL inspection and traffic shaping rules.

Friday, August 1, 2014

Debugging 0x8004005 error in SCCM 2012 R2 OSD deployment

I just hit a most common 0x8004005 error code with my TS (SCCM 2012 R2) on a Lenovo M92z that otherwise runs fine or iMac, MacMini and many other systems including Lenovo laptops.

Note 1: I have designed a relatively simple adapting task sequence that deploys on all makes and models in my institution.

Note 2: Unless you have a good reason to avoid it, I strongly suggest to activate command prompt support. At least in my case, it is perfect for debugging and solving all sorts of issues.

In my case, the error appeared right before Setup Windows. But this does not mean anything. In fact the error itself just means "An error occurred."

In order to resolve the issue, I have used F8 to go to command prompt as soon as the error appears on the deployed computer. 
In C:\_SMSTaskSequence\Logs\smsts.log , you will probably find nothing useful but still you need to check it. Open it using the following command (assuming you are in the right directory): notepad smsts.log .
As a second step, in the X:\Windows\panther\setupact.log we will find what we are looking for... or maybe not??? Depending on the failure, the log file may be located in the following places (check them all even if you found a log file somewhere else!!!):
  • C:\_SMSTaskSequence\Logs\smsts.log
  • X:\Windows\panther\setupact.log
  • X:\Windows\Temp\SMSTSLog\smsts.log
Note 3: If 15 minutes have gone and the SCMM client has completely erased C:\_SMSTaskSequence look in C:\ and in sub-folders for the setupact.log.  

Note 4: If Windows installs but some settings do not get applied use the above mentioned setupact.log with setuperr.log, both in C:\Windows\panther\, to check for errors.

CASE 1

The error reads: Windows Setup could not install one or more boot-critical drivers. To install Windows, make sure that the drivers are valid, and restart the installation.
In this case: the error indicates that Windows failed to import a boot critical driver. On the previous line, the log identifies the driver that was last imported: btmleihd.inf. But what is it and how to find it?
In SCCM --> Drivers, search for a driver with criteria INF file:

It is an Intel Bluetooth driver that windows, for some reason, has considered critical and applicable to this specific system. This driver needs to be disabled or deleted or ... anything you want as long as it does not get installed on the given machine.

CASE 2

If the error occurs before the task sequence starts, check the time on the computer and make sure that there is at least one TS available for that machine. It is possible that the time of the computer is set to let's say 2005 (faulty battery or BIOS/UEFI reset) and all TS are available from 2014. Similarly, the computer may be classified in the wrong category and has no TS associated with it. 

CASE 3

It is possible that the deployment failed for another reason. For example:
  • could not set boot record
  • could not find boot partition
  • etc.

Friday, July 25, 2014

Default User Start Screen

I have basically followed the tutorial here (using PowerShell of course):

http://www.scconfigmgr.com/2013/10/23/modify-the-windows-8-1-start-screen-during-osd-in-configmgr-2012/

Steps:

  1. Deploy OS
  2. Configure the Start Screen as you want on the default admin account
  3. Export the config in a bin file using the command in elevated PowerShell (make sure that Temp folder exists): Export-StartLayout -As Bin -Path C:\Temp\appsFolderLayout.bin
  4. Using a package or any other deployment method of your choice copy this bin to default user profile during OSD. Command line example: 
xcopy appsFolderLayout.bin "%SystemDrive%\Users\Default\AppData\Local\Microsoft\Windows" /Q /Y

Saturday, July 19, 2014

Input Localisation for different systems

As usual, Apple is different from any other brand name computer: their keyboard for french Canadian localisation is different from any other brand. All other standard Canadian french keyboards have a layout called exactly that: Canadian French. Apple french keyboard is Canadian Multilingual Standard. Following is the combined documentation resulting of almost a week of reading and tests on systems to finally figure it all out.

Ideally in a French Cnadian stup environment or in any other setup with multiple default keyboard options, we are looking for the following:
  • Apple:
    • FRA-CMS (French Canadian Multilingual Standard) keyboard
    • ENG-CMS (English Canadian Multilingual Standard) keyboard
  • anything else
    • FRA-CAFR (French Canadian French) keyboard
    • ENG-CMS (English Canadian Multilingual Standard) or ENG-US (English US) keyboard
It could be easily accomplished by changing the settings and deleting the extras that we do not need in the following list. Then, this admin settings can be copied to New user profiles and System/login screen settings.

However, this task seems to be harder to do during the deployment. For the moment, I haven't found any way to do it.
In general, we could, as usual, use our make collection variable to set the Input Locale but how?
  1. Using unattend.xml
  2. Using custom script
  3. Direct registry edit

Codes

Before we start, we have to figure out the codes for each Input Locale:
From Microsoft (technet and msdn), we can get two following sets of info:
  • http://technet.microsoft.com/en-us/library/hh825682.aspx
    • English - Canada
      • Primary --> en-CA: United States - English (1009:00000409)
      • Secondary ---> en-CA: Canadian Multilingual Standard (1009:00011009)
    • French - Canada
      • Primary --> fr-CA: Canadian Multilingual Standard (0c0c:00011009)
      • Secondary ---> en-CA: Canadian Multilingual Standard (1009:00011009)
  • http://msdn.microsoft.com/en-ca/goglobal/bb895996.aspx
    • English_Canadian
      • 1009:00000409,
      • 1009:00011009,
      • 1009:00001009
    • French_Canadian
      • 0c0c:00011009,
      • 0409:00000409
Have you noticed something? Actually there are few things that do not look nice.
  1. French - Canada codes from technet we notice that there is no French Canadian keyboard. Moreover, the secondary keyboard specified is actually an English one?!
  2. The article from MSDN specifies two keyboards for French Canadian and three form English Canadian?
To clear out all this weird things, we have to look at how the codes are defined. The code has two components:  language and keyboard layout separated by a column. Moreover, the combination has to be compatible. That is, we cannot set "anything: anything" and just assume that it will work.
Languages:
  • 0c0c - French (fr)
  • 0409 - English (en) US
  • 1009 - English (en) Canadian
Keyboard layout:
  • 00000409 - US layout
  • 00011009 - Canadian Multilingual Standard layout
  • 00001009 - Who knows??? According to some old post from 2003, it looks like it is French Canadian. We will confirm it in the section 3.

1. Using unattend.xml

This method involves two parts, we could try to use custom collection variables directly in unattend.xml or we could use a custom script to modify the unattend.xml settings during TS. The first option will be covered here and the second will be discussed in the third section.

Unfortunately, I was not able to make custom collection variables work. Defining custom collection variables in TS or in appropriate collection and using them in unattend.xml did not give any results. The variables were not replaced by their respective values. As some have suggested, I removed the line in xml referencing to wim file: no difference. If some one wants to test this method, look at C:\Windows\Panther\UnattendGC\setuperr.txt . If this file has no errors recorded (empty) and you still want to check what was set and how look at setupact.txt in the same location. Somewhere towards the middle of the file setupact.txt you will find the following strings:

In this specific example you may notice that unattend.xml specifies two languages: "0c0c:00001009;0409:00000409" or, according to our codes, "French - Canadian French" and "English - US" inputs. However, the unattend.exe will add four(4) languages! It adds the same languages and in the same order as we see in the first example image of this post. That is, we get:
  • en-CA with CMS
  • fr-CA with CAFR
  • fr-CA with CMS
  • en-US
In this specific example, it looks like the code 0c0c:00001009 has triggered the installation of all 3 keyboards. However, the default keyboard was selected as fr-CA with CAFR because this specific deployment was on a Lenovo laptop. Similar behavior on an iMac with original keyboard gives us fr-CA with CMS as the default keyboard. While it looks like the keyboards are properly selected during deployment, it is still not good enough: users will get mixed up with two different french keyboards.

So lets attempt to deploy the system with no input locale specified. In the case of our Lenovo test system, we get en-CA with CMS and fr-CA with CMS. In the Apple install, we get

2. Using custom script

There are two ways to use a custom script: first we can try to use power-shell to set the input locale or we can try to use a script to set in the unattend.xml the locale. The second method is obviously out of question (see the experiments with unattend.xml). 
The script itself is simple:
Set-WinUserLanguageList -LanguageList fr-CA,en-CA -force
It will work in windows 8 and 8.1 but not earlier. It can be used as part of GPO deployment or in the TS. This is how it looks in the TS:


Unfortunately this methods is not perfect either:
  1. PowerShell script does not accept codes.
  2. It works only for the account it is deployed on
Thus, we are back to the square one: too many keyboards for no reason and no control of what gets set where. Our only solution is to go with a registry edit. Of course, this change will be probably done by a script but it will still be a direct reg change with all the problems it can lead to.

3. Direct registry edit

Probably the best and the worst way to do it but it looks like the only option left. It is the worst because we can kill the system; the best because it looks like the only way to control exactly what keyboard is applied where.
According to the blogpost on http://www.powershellmagazine.com/2014/03/24/set-keyboard-layouts-available-on-logon-screen-in-windows-8-1/ , we can simply copy appropriate codes to appropriate places.
Current user input locale keys are located in HKEY_CURRENT_USER\Keyboard Layout\Preload
Default input locale is located in HKEY_USERS.DEFAULT\Keyboard Layout\Preload key
In the registry, the input locale is supposed to be stored in a hex format (LCIDHex code to be precise). The codes can be found in the already familiar list: http://msdn.microsoft.com/en-ca/goglobal/bb895996.aspx For example, LCIDHex code 0409 is in fact en-US or 0409:00000409 input locale. However... the example below will demonstrate that this is not the case.

In the case of the above mentioned Lenovo install, we can see in the HKEY_CURRENT_USER\Keyboard Layout\Preload the following keys:
In the HKEY_CURRENT_USER\Keyboard Layout\Substitutes we can also find:
For this Lenovo, we do not need fr-CA with CMS input locale. Removing it (using the interface) gives us the following keys and keyboards left:
HKEY_CURRENT_USER\Keyboard Layout\Preload
HKEY_CURRENT_USER\Keyboard Layout\Substitutes


So lets continue and keep only the two keyboards we need. Lets say it is en-US and fr-CA with CAFR keyboard:
HKEY_CURRENT_USER\Keyboard Layout\Preload
HKEY_CURRENT_USER\Keyboard Layout\Substitutes

From the above tests, we can see that the combination of Preload and Substitutes creates the keyboards we use for all systems except Apple make. Notice that:
  • the registry uses the same old codes(not IDs) that we have always delt with
  • the order in the Preload defines the default keyboard (first in the list)
  • the locale that has a secondary/tertiary keyboard is detailed in the Substitutes. For example, en-US (code 409)  has only one keyboard; hence, no entry in the Substitutes. fr-CA (code 0c0c) has many options; therefore, there is a specification of exact language and layout in the Substitutes: 0c0c - French Canadian with 1009 - French Canadian keyboard.

So now we can set this in HKEY_USERS.DEFAULT\Keyboard Layout\Preload key  with the following registry file (or any other script that can suit you): 

Windows Registry Editor Version 5.00

[HKEY_USERS\.DEFAULT\Keyboard Layout]
[HKEY_USERS\.DEFAULT\Keyboard Layout\Preload]
"1"="00000c0c"
"2"="00000409"
[HKEY_USERS\.DEFAULT\Keyboard Layout\Substitutes]
"00000c0c"="00001009"
[HKEY_USERS\.DEFAULT\Keyboard Layout\Toggle]

Now, how can we set it in the login screen? According to microsoft kb (http://support.microsoft.com/kb/243330) there are a few security IDs that we can identify. We can also use a full registry search to find all related settings. After all, we will get the following additional registry keys to set:
[HKEY_USERS\S-1-5-18\Keyboard Layout]
[HKEY_USERS\S-1-5-18\Keyboard Layout\Preload]
"1"="00000c0c"
"2"="00000409"
[HKEY_USERS\S-1-5-18\Keyboard Layout\Substitutes]
"00000c0c"="00001009"
[HKEY_USERS\S-1-5-18\Keyboard Layout\Toggle]

[HKEY_USERS\S-1-5-19\Keyboard Layout]
[HKEY_USERS\S-1-5-19\Keyboard Layout\Preload]
"1"="00000c0c"
"2"="00000409"
[HKEY_USERS\S-1-5-19\Keyboard Layout\Substitutes]
"00000c0c"="00001009"
[HKEY_USERS\S-1-5-19\Keyboard Layout\Toggle]

[HKEY_USERS\S-1-5-20\Keyboard Layout]
[HKEY_USERS\S-1-5-20\Keyboard Layout\Preload]
"1"="00000c0c"
"2"="00000409"
[HKEY_USERS\S-1-5-20\Keyboard Layout\Substitutes]
"00000c0c"="00001009"
[HKEY_USERS\S-1-5-20\Keyboard Layout\Toggle]

Note that 1-5-18 is a Local System account used by the OS; 1-5-19 is NT Authority as a Local Service account and 1-5-20 is NT Authority as Network Service account. Following the same strategy, we get the registry keys for Apple type of computers:
Windows Registry Editor Version 5.00

[HKEY_USERS\.DEFAULT\Keyboard Layout]
[HKEY_USERS\.DEFAULT\Keyboard Layout\Preload]
"2"="00001009"
"1"="00000c0c"
[HKEY_USERS\.DEFAULT\Keyboard Layout\Substitutes]
"00001009"="00011009"
"00000c0c"="00011009"
[HKEY_USERS\.DEFAULT\Keyboard Layout\Toggle]

[HKEY_USERS\S-1-5-18\Keyboard Layout]
[HKEY_USERS\S-1-5-18\Keyboard Layout\Preload]
"2"="00001009"
"1"="00000c0c"
[HKEY_USERS\S-1-5-18\Keyboard Layout\Substitutes]
"00001009"="00011009"
"00000c0c"="00011009"
[HKEY_USERS\S-1-5-18\Keyboard Layout\Toggle]

[HKEY_USERS\S-1-5-19\Keyboard Layout]
[HKEY_USERS\S-1-5-19\Keyboard Layout\Preload]
"2"="00001009"
"1"="00000c0c"
[HKEY_USERS\S-1-5-19\Keyboard Layout\Substitutes]
"00001009"="00011009"
"00000c0c"="00011009"
[HKEY_USERS\S-1-5-19\Keyboard Layout\Toggle]

[HKEY_USERS\S-1-5-20\Keyboard Layout]
[HKEY_USERS\S-1-5-20\Keyboard Layout\Preload]
"2"="00001009"
"1"="00000c0c"
[HKEY_USERS\S-1-5-20\Keyboard Layout\Substitutes]
"00001009"="00011009"
"00000c0c"="00011009"
[HKEY_USERS\S-1-5-20\Keyboard Layout\Toggle]

However, all this work cannot help us if we do not find the way to edit the default user profile saved in ntuser.dat. As aresult, we will have to use some script. In my case, I chose to convert all of the above keys into a cmd line script that will edit the default user profile. At the same time, I could use the script to do more changes (in the default profile or any other profile for that matter): see http://stealthpuppy.com/customize-the-windows-default-profile/ and http://technet.microsoft.com/en-us/library/cc742162.aspx.
For those who like more a VB style approach look here: http://micksmix.wordpress.com/2012/01/13/update-a-registry-key-for-all-users-on-a-system/.
For a PowerShell approach, look here: http://www.powershellmagazine.com/2014/03/24/set-keyboard-layouts-available-on-logon-screen-in-windows-8-1/.

It finally looks like this:

Other
Apple
@ECHO OFF
REM Load the default profile hive
SET HKEY=HKU\Default
REG LOAD %HKEY% %SystemDrive%\Users\Default\NTUSER.DAT

REM Set keyboard locale for Default User
REG ADD "%HKEY%\Keyboard Layout" /f
REG ADD "%HKEY%\Keyboard Layout\Preload" /f
REG ADD "%HKEY%\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "%HKEY%\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00000409" /f
REG ADD "%HKEY%\Keyboard Layout\Substitutes" /f
REG ADD "%HKEY%\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00001009" /f

REM Set keyboard locale for .default
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Preload" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00000409" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00001009" /f

REM Set keyboard locale for S-1-5-18
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Preload" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00000409" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00001009" /f

REM Set keyboard locale for S-1-5-19
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Preload" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00000409" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00001009" /f

REM Set keyboard locale for S-1-5-20
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Preload" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00000409" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00001009" /f

REM Set keyboard locale for Current User
REG ADD "HKEY_CURRENT_USER\Keyboard Layout" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Preload" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00000409" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00001009" /f

REM Unload the default profile hive
REG UNLOAD %HKEY%
@ECHO OFF
REM Load the default profile hive
SET HKEY=HKU\Default
REG LOAD %HKEY% %SystemDrive%\Users\Default\NTUSER.DAT

REM Set keyboard locale for Default User
REG ADD "%HKEY%\Keyboard Layout" /f
REG ADD "%HKEY%\Keyboard Layout\Preload" /f
REG ADD "%HKEY%\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "%HKEY%\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00001009" /f
REG ADD "%HKEY%\Keyboard Layout\Substitutes" /f
REG ADD "%HKEY%\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00011009" /f
REG ADD "%HKEY%\Keyboard Layout\Substitutes" /v "00001009" /t REG_SZ /d "00011009" /f

REM Set keyboard locale for .default
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Preload" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00001009" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00011009" /f
REG ADD "HKEY_USERS\.DEFAULT\Keyboard Layout\Substitutes" /v "00001009" /t REG_SZ /d "00011009" /f

REM Set keyboard locale for S-1-5-18
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Preload" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00001009" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00011009" /f
REG ADD "HKEY_USERS\S-1-5-18\Keyboard Layout\Substitutes" /v "00001009" /t REG_SZ /d "00011009" /f

REM Set keyboard locale for S-1-5-19
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Preload" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00001009" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00011009" /f
REG ADD "HKEY_USERS\S-1-5-19\Keyboard Layout\Substitutes" /v "00001009" /t REG_SZ /d "00011009" /f

REM Set keyboard locale for S-1-5-20
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Preload" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00001009" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00011009" /f
REG ADD "HKEY_USERS\S-1-5-20\Keyboard Layout\Substitutes" /v "00001009" /t REG_SZ /d "00011009" /f

REM Set keyboard locale for Current User
REG ADD "HKEY_CURRENT_USER\Keyboard Layout" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Preload" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Preload" /v "1" /t REG_SZ /d "00000c0c" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Preload" /v "2" /t REG_SZ /d "00001009" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Substitutes" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Substitutes" /v "00000c0c" /t REG_SZ /d "00011009" /f
REG ADD "HKEY_CURRENT_USER\Keyboard Layout\Substitutes" /v "00001009" /t REG_SZ /d "00011009" /f

REM Unload the default profile hive
REG UNLOAD %HKEY%
Each script is part of a package and is configured as an application in SCCM. This will give me an ability to deploy the script to any machine at any time. Note however, that if you choose to deploy the script as an application, the changes in the registry may not be immediately visible to active user: log-off / log-in or a restart sequence is usually a good idea.

TO BE CONTINUED

Dynamicly apply drivers in SCCM 2012 R2

As I have mentioned before, we have Apple, Lenovo, HP and Dell brands in the domain. We will use a custom collection variable make to differentiate between different brands and apply appropriate drivers.

In general, we have imported drivers with appropriate labels and everything to ensure that we can control the deployment and future cleanup in a granular manner. However, during the deployment, unless there is a compatibility issue, I prefer to use more broad categories such as Apple, Lenovo etc instead of going into lots of details (ex.: Lenovo 10AF, Lenovo 10AD, Apple iMac 9.1 etc.). In any case, SCCM will install only the best matched drivers from the categories selected in this step.

In a TS it looks like this:

The config for Apple Device drivers is the following:


For Lenovo, HP and others, we have to choose the appropriate category and appropriate value for make variable. For Unknown System it looks like this:


Note that in the case of unknown system, the computer will not be part of any predefined dynamic collection. Therefore, the custom variable will not exist in this case. 


Dynamic partitioning of HDD in SCCM 2012 R2

In general, the default partitioning in SCCM 2012 R2 TS can suit most deployments but not Apple!
In fact, if we want to keep MAC partition and have dual boot with rEFIt or any other config, we have to somehow format the partition and leave everything else intact.

To do this, we will use Apple Deploy Studio to create/format partitions and deploy OS X. Then we will format the partition in TS.



Notice that the condition has a task sequence variable make equals "Apple". This variables comes from a dynamic collection Apple that I have defined in a previous post on OEM branding.

The script DiskPartMac.txt contains:
SELECT disk 0
SELECT partition 3
FORMAT FS=NTFS LABEL="Win8" QUICK NOERR OVERRIDE
ASSIGN LETTER=C NOERR
ACTIVE
EXIT
The script can be adjusted and changed to take into account any number of partitions. For now, it takes into account one MAC OS X and one windows partition: remember that there is always an EFI partition + MAC+ Windows. However, it is important to note that partitions themselves are easier to create by the Deploy Studio: we were never able to make it work on opposite direction.
  
Of course, we will add some conditions to the default HD partitionning in the TS. For both, BIOS and UEFI partitioning, we will add the following: make not equals "Apple". For example, conditions for formatting in BIOS boot will look like this:



Eject CD script

As part of our setup, we have to deploy some iMacs and MacMinis all of them do not have PXE. Moreover, we prefer a manually controlled deployment method and do not want to use required deployment.

My colleague has designed a CD that loads WinPE. However, the CD needs to be ejected ASAP. here is the small script to do it in CMD:
"Eject CD\eject.exe"

It uses an eject.exe that is part of a small package of scripts that we have created for all our TS needs. If I remember correctly, this exe can be copied from MDT deployment share Scripts folder.

In the TS, it looks like this:


Dynamic OEM Computer Model

One of the nice touches of proper and detailed TS config is the appropriate use of OEM information. In fact, it is sometimes useful to have OEM models setup according to the hardware specs.

The script is really simple:
REG ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\OEMInformation /V Model /T REG_SZ /D "%make% %model%" /F




It leverages two custom variables %make% and %model% defined in the appropriate collection. For example, Apple collection is created with a query
select SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System inner join SMS_G_System_COMPUTER_SYSTEM on SMS_G_System_COMPUTER_SYSTEM.ResourceId = SMS_R_System.ResourceId where SMS_G_System_COMPUTER_SYSTEM.Manufacturer = "Apple Inc."
and has a variable make=Apple

While collection MacMini 5.1 is using a query
select SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System inner join SMS_G_System_COMPUTER_SYSTEM on SMS_G_System_COMPUTER_SYSTEM.ResourceID = SMS_R_System.ResourceId where SMS_G_System_COMPUTER_SYSTEM.Model = "Macmini5,1"
and has a variable model=MacMini 5,1 (Mid 2011)

The combination of both variables (during deployment to the above mentioned model and make) gives a final script for this model:
REG ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\OEMInformation /V Model /T REG_SZ /D "Apple MacMini 5,1 (Mid 2011)" /F

Of course all this work for make and model in OEM branding do not make sense. We are using them extensively throughout the TS: in HDD formatting, driver injection and many more other places.

Unstable Aerohive behavior

Recently I had some issues with the entire hive: the APs where restarting randomly and the users were unable to register. immediate hot fix was to restart the units but it was good only for a few days or a week max.

First, diagnostics: the issue we had is completely silent and hidden. No error messages or any other problems show in the logs both on the APs and in the manager. The only warning sign is frequent random restarts of APs: roughly every 3-4 days a few units will restart. Eventually, users that try to authenticate will be refused connection and those who are already authenticated will remain connected.

While working with Aerohive, we found out that it could be due to WIPS (). It is possible that WIPS will force some sort of kernel panic effectively inducing a unit restart. By disabling WIPS, we have extended the restart time and the interval between authentication problems. However the issue was not fixed completely: once a week or a bit more we had to restart everything again.

The solution? We haven't found one. Luckily, a new version came out (6.1r6a). At the moment, it looks like a full upgrade has fixed the issues: no restarts or connection issues for the past week.