2015年2月28日土曜日

How to download a file with Scrapy?

1.Way 1

Save the pdf in the spider callback:

def parse_listing(self, response):      # ... extract pdf urls      for url in pdf_urls:          yield Request(url, callback=self.save_pdf)    def save_pdf(self, response):      path = self.get_path(response.url)      with open(path, "wb") as f:          f.write(response.body)

2.Way 2

Do it in pipeline :

# in the spider  def parse_pdf(self, response):      i = MyItem()      i['body'] = response.body      i['url'] = response.url      # you can add more metadata to the item      return i    # in your pipeline  def process_item(self, item, spider):      path = self.get_path(item['url'])      with open(path, "wb") as f:          f.write(item['body'])      # remove body and add path as reference      del item['body']      item['path'] = path      # let item be processed by other pipelines. ie. db store      return item

3.Way 3

Use the filepiple(https://groups.google.com/forum/print/msg/scrapy-users/kzGHFjXywuY/O6PIhoT3thsJ)

1) download https://raw.github.com/scrapy/scrapy/master/scrapy/contrib/pipeline/files.py
and save it somewhere in your Scrapy project,
let's say at the root of your project (but that's not the best location…)

yourproject/files.py

2) then, enable this pipeline by adding this to your settings.py

ITEM_PIPELINES = [      'yourproject.files.FilesPipeline',  ]  FILES_STORE = '/path/to/yourproject/downloads'

FILES_STORE needs to point to a location where Scrapy can write (create it beforehand)

3) add 2 special fields to your item definition

    file_urls = Field()      files = Field()

4) in your spider, when you have an URL for a file to download,
add it to your Item instance before returning it

...      myitem = YourProjectItem()      ...      myitem["file_urls"] = ["http://www.example.com/somefileiwant.csv"]      yield myitem

5) run your spider and you should see files in the FILES_STORE folder

Here's an example that download a few files from the IETF website

the scrapy project is called "filedownload"

items.py looks like this:

from scrapy.item import Item, Field    class FiledownloadItem(Item):      file_urls = Field()      files = Field()

this is the code for the spider:

from scrapy.spider import BaseSpider  from filedownload.items import FiledownloadItem    class IetfSpider(BaseSpider):      name = "ietf"      allowed_domains = ["ietf.org"]      start_urls = (          'http://www.ietf.org/',          )        def parse(self, response):          yield FiledownloadItem(              file_urls=[                  'http://www.ietf.org/images/ietflogotrans.gif',                  'http://www.ietf.org/rfc/rfc2616.txt',                  'http://www.rfc-editor.org/rfc/rfc2616.ps',                  'http://www.rfc-editor.org/rfc/rfc2616.pdf',                  'http://tools.ietf.org/html/rfc2616.html',              ]          )      

Reference:

https://stackoverflow.com/questions/7123387/scrapy-define-a-pipleine-to-save-files

https://groups.google.com/forum/#!msg/scrapy-users/kzGHFjXywuY/O6PIhoT3thsJ

2015年2月27日金曜日

Use PowerShell and Regular Expressions to Search Binary Data

Regular expressions are great for searching for patterns in text. This should come as no surprise because this is what they were designed to do. However, what if you want to perform regular expressions on binary data? For example, say there is a particular sequence of bytes in a file that don't map to printable characters? Doing this in Windows PowerShell poses two unique challenges:

Regular expressions in Windows PowerShell and .NET only work on strings, not on byte arrays.
Depending on the character encoding used, not all bytes translate to characters and then back into their same byte value.
Hexadecimal representation with regular expressions

Before tackling these challenges, it would be useful to cover how to search for bytes in their hexadecimal representation using regular expressions.

For example, the word 'Hello' is comprised of five characters, each with a corresponding numeric value. Let's use Windows PowerShell to see the hexadecimal values that make up the string 'Hello':

$StringBytes = [System.Text.Encoding]::ASCII.GetBytes('Hello')

[System.BitConverter]::ToString($StringBytes)

The first line converts 'Hello' to a byte array. Next, the BitConverter class ToString method prints out the hexadecimal digits of the byte array as a string: '48-65-6C-6C-6F'. Armed with this knowledge, let's search for the word 'Hello' by using only hexadecimal:

if ('Hello, cruel world!' -match '\x48\x65\x6C{2}\x6F')

{

$Matches[0]

}

Searching for hexadecimal bytes with a regular expression requires that you prepend '\x' to the byte. The regular expression here is searching for an 'H', 'e', exactly two 'l's, and an 'o'.

Scanning for other bytes

Now that we have a foundation for using regular expressions to scan for a sequence of bytes, let's begin analyzing bytes that don't normally translate to pretty characters. As an example, I recently found myself wanting to scan 32-bit executables for hotpatchable functions.

Many 32-bit Windows functions begin with the following two-byte assembly language instruction: MOV EDI, EDI. This instruction doesn't serve any useful purpose other than serving as a placeholder for a Microsoft hotpatch.

Hotpatching allows patches to be installed while a process is running. This functionality is essential when applying patches to servers that cannot be rebooted immediately. This blog post by Raymond Chen explains this concept in more detail: Why do Windows functions all begin with a pointless MOV EDI, EDI instruction?

But basically, I'm interested in finding a sequence of five '\xCC' or '\x90' bytes followed by '\x8B\xFF'. This regular expression would look like this: '[\xCC\x90]\x8B\xFF'.

Using 28591 encoding

If we're going to scan for any arbitrary sequence of bytes with values between 0 and 255, we have to be sure that each character of a string that represents the binary data maps back to its respective byte value. Typically, in Windows PowerShell, you would read a file as a raw string with Get-Content ?Raw and use the -Encoding flag to specify the character encoding to be used (for example, Unicode, ASCII, UTF7, or UTF8).

Unfortunately, none of the encoding schemes that are allowed in Get-Content provide a one-to-one mapping of characters back to its respective byte value. There is a magic encoding scheme that does, however: ISO-8859-1 (Codepage: 28591).

To create an encoder object for an arbitrary code page, you can use the GetEncoding static method of the .NET Encoding class. An example of this is shown here:

$Encoder = [System.Text.Encoding]::GetEncoding(28591)

We cannot provide this encoder to the Get-Content cmdlet, so we have to use some .NET to read a file as a string using our special encoding. I wrote a filter to perform the needed operation. This filter is shown here, and I uploaded the full script to the Script Center Repository: ConvertTo-String.

ConvertTo-String filter:

filter ConvertTo-String

{

<#

.SYNOPSIS

Converts the bytes of a file to a string.

.DESCRIPTION

ConvertTo-String converts the bytes of a file to a string that has a

1-to-1 mapping back to the file's original bytes. ConvertTo-String is

useful for performing binary regular expressions.

.PARAMETER Path

Specifies the path to a file.

.EXAMPLE

PS C:\>$BinaryString = ConvertTo-String C:\Windows\SysWow64\kernel32.dll

PS C:\>$HotpatchableRegex = [Regex] '[\xCC\x90]{5}\x8B\xFF'

PS C:\>$HotpatchableRegex.Matches($BinaryString)



Description

-----------

Converts kernel32.dll into a string. A binary regular expression is

then performed on the string searching for a hotpatchable code

sequence - i.e. 5 nop/int3 followed by a mov edi, edi instruction.

#>

[OutputType([String])]

Param (

[Parameter( Mandatory = $True,

Position = 0,

ValueFromPipeline = $True )]

[ValidateScript( { -not (Test-Path $_ -PathType Container) } )]

[String]

$Path

)



$Stream = New-Object IO.FileStream -ArgumentList (Resolve-Path $Path), 'Open', 'Read'



# Note: Codepage 28591 returns a 1-to-1 char to byte mapping

$Encoding = [Text.Encoding]::GetEncoding(28591)

$StreamReader = New-Object IO.StreamReader -ArgumentList $Stream, $Encoding

$BinaryText = $StreamReader.ReadToEnd()

$StreamReader.Close()

$Stream.Close()

Write-Output $BinaryText

}

Understanding the ConvertTo-String filter

ConvertTo-String relies upon the FileStream object that accepts any encoder. A StreamReader object is then used to take advantage of its ReadToEnd method, which reads a stream to the end and outputs a string using the encoding scheme specified.

Let's wrap things up and use this function to find the index of each hotpatchable function in the 32-bit version of kernel32.dll:

$BinaryString = ConvertTo-String C:\Windows\SysWow64\kernel32.dll

$HotpatchableRegex = [Regex] '[\xCC\x90]{5}\x8B\xFF'

$HotpatchMatches = $HotpatchableRegex.Matches($BinaryString)

$MatchCount = $HotpatchMatches.Count

Write-Host "Total number of matches: $MatchCount"

# Print the index (in hexidecimal) of each MOV EDI, EDI instruction

$HotpatchMatches |

ForEach-Object { "0x$(($_.Index + 5).ToString('X8'))" }

Use PowerShell to Interact with the Windows API

Reflection gives a programmer the ability to perform type introspection on code. The most common form of type introspection you would perform in Windows PowerShell would be using the Get-Member cmdlet, which enables you to discover the methods and properties of an object. In the last post, we also used reflection to find a non-public method that implemented the kernel32.dll CopyFile function. In this post, we will be using reflection to generate code dynamically. This concept is known as metaprogramming.

Steps to define a dynamic method

To define a dynamic method that will call the CopyFile method in kernel32.dll, the following steps are needed:

Define a dynamic assembly. Recall that the assembly is the container for modules, types, and members.
Define the dynamic assembly in your current AppDomain. Think of the AppDomain as your Windows PowerShell session. It's the environment in which our method is going to execute.
Define a dynamic module. A module is a container for types and their members.
Define a dynamic type. A type (that is, class) is a container for members (methods, properties, nested types, fields, events, and constructors).
Define our dynamic method. Here we specify attributes (Public and Static) and the method's parameters and return type.
Manually build a DllImport attribute. The result will be the equivalent of the following C# attribute:
[DllImport("kernel32.dll", SetLastError = true, PreserveSig = true, CallingConvention = CallingConvention.WinApi, CharSet = CharSet.Unicode)]
Apply the custom DllImport attribute to the dynamic method.
Call the CreateType method to bake everything together and make our method available to our Windows PowerShell session.
As you can see, this is not a trivial process. This process involves performing tasks that Add-Type and the C# compiler would typically take care of on your behalf.

So you may be asking now, "Why the heck would I want to go through all this trouble just to call a function?" The reason I typically do this is twofold:

I need to maintain a minimal forensic footprint while executing my script. I don't want to invoke the C# compiler and write temp files to disk.
The Windows API function I want is not present in the portion of the .NET Framework loaded by my current Windows PowerShell session.
To see all of this in action, I wrote another implementation of the Copy-RawItem function that uses reflection. The complete text of this function is available in the Script Center Repository: Copy-RawItem (Reflection Version).

Copy-RawItem ? Reflection Version:

function Copy-RawItem

{

<#

.SYNOPSIS

Copies a file from one location to another including files contained within DeviceObject paths.

.PARAMETER Path

Specifies the path to the file to copy.

.PARAMETER Destination

Specifies the path to the location where the item is to be copied.

.PARAMETER FailIfExists

Do not copy the file if it already exists in the specified destination.

.OUTPUTS

None or an object representing the copied item.

.EXAMPLE

Copy-RawItem '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Windows\System32\config\SAM' 'C:\temp\SAM'

#>

[CmdletBinding()]

[OutputType([System.IO.FileSystemInfo])]

Param (

[Parameter(Mandatory = $True, Position = 0)]

[ValidateNotNullOrEmpty()]

[String]

$Path,

[Parameter(Mandatory = $True, Position = 1)]

[ValidateNotNullOrEmpty()]

[String]

$Destination,

[Switch]

$FailIfExists

)

# Create a new dynamic assembly. An assembly (typically a dll file) is the container for modules

$DynAssembly = New-Object System.Reflection.AssemblyName('Win32Lib')

# Define the assembly and tell is to remain in memory only (via [Reflection.Emit.AssemblyBuilderAccess]::Run)

$AssemblyBuilder = [AppDomain]::CurrentDomain.DefineDynamicAssembly($DynAssembly, [Reflection.Emit.AssemblyBuilderAccess]::Run)

# Define a new dynamic module. A module is the container for types (a.k.a. classes)

$ModuleBuilder = $AssemblyBuilder.DefineDynamicModule('Win32Lib', $False)

# Define a new type (class). This class will contain our method - CopyFile

# I'm naming it 'Kernel32' so that you will be able to call CopyFile like this:

# [Kernel32]::CopyFile(src, dst, FailIfExists)

$TypeBuilder = $ModuleBuilder.DefineType('Kernel32', 'Public, Class')

# Define the CopyFile method. This method is a special type of method called a P/Invoke method.

# A P/Invoke method is an unmanaged exported function from a module - like kernel32.dll

$PInvokeMethod = $TypeBuilder.DefineMethod(

'CopyFile',

[Reflection.MethodAttributes] 'Public, Static',

[Bool],

[Type[]] @([String], [String], [Bool]))

#region DllImportAttribute

# Set the equivalent of: [DllImport(

# "kernel32.dll",

# SetLastError = true,

# PreserveSig = true,

# CallingConvention = CallingConvention.WinApi,

# CharSet = CharSet.Unicode)]

# Note: DefinePInvokeMethod cannot be used if SetLastError needs to be set

$DllImportConstructor = [Runtime.InteropServices.DllImportAttribute].GetConstructor(@([String]))

$FieldArray = [Reflection.FieldInfo[]] @(

[Runtime.InteropServices.DllImportAttribute].GetField('EntryPoint'),

[Runtime.InteropServices.DllImportAttribute].GetField('PreserveSig'),

[Runtime.InteropServices.DllImportAttribute].GetField('SetLastError'),

[Runtime.InteropServices.DllImportAttribute].GetField('CallingConvention'),

[Runtime.InteropServices.DllImportAttribute].GetField('CharSet')

)

$FieldValueArray = [Object[]] @(

'CopyFile',

$True,

$True,

[Runtime.InteropServices.CallingConvention]::Winapi,

[Runtime.InteropServices.CharSet]::Unicode

)

$SetLastErrorCustomAttribute = New-Object Reflection.Emit.CustomAttributeBuilder(

$DllImportConstructor,

@('kernel32.dll'),

$FieldArray,

$FieldValueArray)

$PInvokeMethod.SetCustomAttribute($SetLastErrorCustomAttribute)

Use PowerShell to Interact with the Windows API

For most people, the Add-Type method will always be sufficient. However, for people like me who write security and forensics tools, where a minimal forensic footprint is necessary, Add-Type doesn't cut it because it writes temporary files to disk and calls csc.exe to compile C# code. So if having a minimal forensic footprint is a necessity for your script (or if you like to do things the hard way), there are other methods of calling Windows API functions.

A large portion of the .NET Framework is actually built on the Windows API. It's just not exposed to you publicly. Pulling out the Win API functions that are used by the .NET Framework requires a basic understanding of the internal layout of .NET.

There are several tools available for exploring the .NET Framework, including these:

.NET Reflector 8
ILSpy .NET Decompiler
What you need to know for our purposes is that Windows API function calls in .NET are usually non-public (that is, private), static methods that have a DllImport attribute associated with them. To help us dig into the .NET Framework, I wrote a helper function called Find-WinAPIFunction. (The script is also available in the Script Center Repository.) It searches the loaded modules in a Windows PowerShell session for a reference to a private Windows API function.

Find-WinAPIFunction function:

function Find-WinAPIFunction

{

<#

.SYNOPSIS

Searches all loaded assemblies in your PowerShell session for a

Windows API function.

.PARAMETER Module

Specifies the name of the module that implements the function. This

is typically a system dll (e.g. kernel32.dll).

.PARAMETER FunctionName

Specifies the name of the function you're searching for.

.OUTPUTS

[System.Reflection.MethodInfo]

.EXAMPLE

Find-WinAPIFunction kernel32.dll CopyFile

#>

[CmdletBinding()]

[OutputType([System.Reflection.MethodInfo])]

Param

(

[Parameter(Mandatory = $True, Position = 0)]

[ValidateNotNullOrEmpty()]

[String]

$Module,

[Parameter(Mandatory = $True, Position = 1)]

[ValidateNotNullOrEmpty()]

[String]

$FunctionName

)

[System.AppDomain]::CurrentDomain.GetAssemblies() |

ForEach-Object { $_.GetTypes() } |

ForEach-Object { $_.GetMethods('NonPublic, Public, Static') } |

ForEach-Object { $MethodInfo = $_; $_.GetCustomAttributes($false) } |

Where-Object {

$MethodInfo.Name.ToLower() -eq $FunctionName.ToLower() -and

$_.Value -eq $Module

} | ForEach-Object { $MethodInfo }

}

The Find-WinAPIFunction function works by drilling down through all of the loaded assemblies in a Windows PowerShell session. To understand what's going on, think of your Windows PowerShell session as a series of containers. It starts with an AppDomain. Each Windows PowerShell session has a default AppDomain, which you can think of as an execution sandbox.

Within an AppDomain, there are multiple loaded assemblies. An assembly is a container for a module, and it is typically a DLL, such as mscorlib.dll or System.dll. Within assemblies, are modules that are containers for types (that is, classes).

Finally, a type is a container for members. Members consist of methods, properties, fields, events, nested types, and constructors. The concept of members should be familiar to those who are familiar with the Get-Member cmdlet. The following diagram may help illustrate this concept of compartmentalization more clearly.

Image of compartments

Find-WinAPIFunction starts by iterating through all assemblies in the current AppDomain. It then iterates through all types within those assemblies, ultimately looking for the method you're searching for.

Image of command output

Now that we know that a CopyFile method exists, we know that we can use it in Windows PowerShell. However, rather than searching for it each time with Find-WinAPIFunction, let's pull out some information that will allow us to quickly get a reference to our target CopyFile method:

Image of command output

As can be seen in this screenshot, we need the name of the assembly and type that contains the CopyFile method. These are respectively mscorlib.dll and Microsoft.Win32.Win32Native.

Now let's pull everything together with another implementation of the Copy-RawItem function, which made its debut in Part 1 of this series. This time, we have the Private .NET Method version, which is also available in the Script Center Repository.

Copy-RawItem Private .NET Method version:

function Copy-RawItem

{

<#

.SYNOPSIS

Copies a file from one location to another including files contained within DeviceObject paths.

.PARAMETER Path

Specifies the path to the file to copy.

.PARAMETER Destination

Specifies the path to the location where the item is to be copied.

.PARAMETER FailIfExists

Do not copy the file if it already exists in the specified destination.

.OUTPUTS

None or an object representing the copied item.

.EXAMPLE

Copy-RawItem '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Windows\System32\config\SAM' 'C:\temp\SAM'

#>

[CmdletBinding()]

[OutputType([System.IO.FileSystemInfo])]

Param (

[Parameter(Mandatory = $True, Position = 0)]

[ValidateNotNullOrEmpty()]

[String]

$Path,

[Parameter(Mandatory = $True, Position = 1)]

[ValidateNotNullOrEmpty()]

[String]

$Destination,

[Switch]

$FailIfExists

)

# Get a reference to the internal method - Microsoft.Win32.Win32Native.CopyFile()

$mscorlib = [AppDomain]::CurrentDomain.GetAssemblies() | ? {$_.Location -and ($_.Location.Split('\')[-1] -eq 'mscorlib.dll')}

$Win32Native = $mscorlib.GetType('Microsoft.Win32.Win32Native')

$CopyFileMethod = $Win32Native.GetMethod('CopyFile', ([Reflection.BindingFlags] 'NonPublic, Static'))

# Perform the copy

$CopyResult = $CopyFileMethod.Invoke($null, @($Path, $Destination, ([Bool] $PSBoundParameters['FailIfExists'])))

$HResult = [System.Runtime.InteropServices.Marshal]::GetLastWin32Error()

if ($CopyResult -eq $False -and $HResult -ne 0)

{

# An error occured. Display the Win32 error set by CopyFile

throw ( New-Object ComponentModel.Win32Exception )

}

else

{

Write-Output (Get-ChildItem $Destination)

}

}

This version of Copy-RawItem provides identical functionality as the Add-Type version, but it implements the private method extraction technique described in this post. The function starts by getting a reference to mscorlib.dll, which is the assembly that contains the CopyFile method. It then gets a reference to the type that contains the Microsoft.Win32.Win32Native method by calling the GetType method. Finally, it gets a reference to the CopyFile method by calling the GetMethod method, specifying that a NonPublic, Static method is requested.

The technique described requires a bit more knowledge of the layout of the .NET Framework. However, armed with this knowledge, you will be able to pull internal functionality into Windows PowerShell, which wouldn't otherwise be available to you.

In the last and final post of the series, we'll dig into .NET internals even more and see how to use reflection to dynamically build a method that calls Windows API functions.

The two scripts used today are available in the Script Center Repository:

How to call .NET and Win32 methods from PowerShell

Every once in a while I run into a problem that I just don't know how to solve using PowerShell directly. Fortunately, PowerShell has great interoperability with other .NET code, so if you're already familiar with VB.NET or C# you'll never be lost.
One of my "bad habits" is to use .NET methods instead of native PowerShell cmdlets. Even now I still find myself using [System.IO.File]::Exists() instead of Test-Path; it's just what pops into my head when I think about that task. I call this a bad habit because some of the folks I interact with aren't C# developers and don't intuitively understand what's going on when there is inline .NET calls all over the place. Here's an example of what I'm talking about. Both these functions do the same thing; depending on your background you may gravitate towards one or the other. Where possible, I recommend using the native PowerShell cmdlet as it'll help other PowerShell users understand what's going on.
# Native PowerShell version
if ((test-path c:\windows\notepad.exe) -eq $true)
{
# do something interesting
$PSFileInfo = Get-ItemProperty c:\windows\notepad.exe
echo "PSFileInfo"
$PSFileInfo
}
# .NET Version
if ([System.IO.File]::Exists("C:\windows\notepad.exe") -eq $true)
{
# do something interesting
$NETFileInfo = New-Object System.IO.FileInfo -ArgumentList "C:\windows\notepad.exe"
echo "NETFileInfo"
$NETFileInfo
}
This style of inline .NET call is pretty convenient when you only need one or two static methods, but what about something more complex? What if you need to call a Win32 API? Well, you're in luck, because PowerShell provides a method to get you on your way. You can use the PowerShell SDK to build a custom cmdlet, but is there another way? Yes, there is!
Let's say that you're building a Windows Troubleshooting Pack that's designed to resolve problems with your device. One of the manual troubleshooting steps you're likely to perform is to check the device in Device Manager for errors, then uninstall and reinstall the device. In order to do this in your troubleshooter you need to be able to:
Identify the device
Get its state (ConfigManagerErrorCode for those familiar with accessing devices using WMI)
Uninstall the device
Reinstall the device
In most cases you'll be able to do the first two tasks directly from PowerShell by using WMI. Uninstalling and reinstalling the device is a bit more complicated, you'll need to call some Win32 APIs to get that done. So let's take a look at how that works in PowerShell and the Windows Troubleshooting Platform. Conceptually, the process looks like this:
Identify the Win32 APIs you need to perform this task
Create a C#/VB.NET class that wraps the Win32 APIs
Bring your wrapper class into PowerShell
Call the methods defined in your wrapper class from PowerShell
Best Practice: While you can do all the steps in this example in Notepad or the PowerShell ISE, I strongly recommend using Visual Studio to create your C#/VB.NET class when doing this for real. Visual Studio will give you access to IntelliSense and all sorts of other features that'll make your life easier. If you don't have Visual Studio you can download the Express edition of your favorite language for free.
I'm going to skip the uninstall step and focus on the reinstall. The concept related to accessing Win32 APIs is identical, but uninstall requires us to do steps one and two first. For the purposes of this example we can manually do steps 1-3 and code only the reinstall.
Step 1: Identify the Win32 APIs you need to perform this task
MSDN is your friend here. In this case we need 3 Win32 APIs:
CM_Locate_DevNode_Ex
CM_Reenumerate_DevNode_Ex
CMP_WaitNoPendingInstallEvents
Step 2: Create a C#/VB.NET class that wraps the Win32 APIs
This is where you'll spend most of your time and why I'd recommend doing this work inside Visual Studio where you have great debugging tools. To save time I've included the entire wrapper source below in C#. Paste this into Visual Studio, save, and do a quick build to make sure there aren't any errors.
using System.Runtime.InteropServices;
using System;

namespace mattbie.examples.devices
{
// These are the native win32 methods that we require
internal static class NativeMethods
{
[DllImport("cfgmgr32.dll", SetLastError = true, EntryPoint = "CM_Locate_DevNode_Ex", CharSet = CharSet.Auto)]
public static extern UInt32 CM_Locate_DevNode_Ex(ref UInt32 DevInst, IntPtr DeviceID, UInt64 Flags, IntPtr Machine);

[DllImport("cfgmgr32.dll", SetLastError = true, EntryPoint = "CM_Reenumerate_DevNode_Ex", CharSet = CharSet.Auto)]
public static extern UInt32 CM_Reenumerate_DevNode_Ex(UInt32 DevInst, UInt64 Flags, IntPtr Machine);

[DllImport("cfgmgr32.dll", SetLastError = true, EntryPoint = "CMP_WaitNoPendingInstallEvents", CharSet = CharSet.Auto)]
public static extern UInt32 CMP_WaitNoPendingInstallEvents(UInt32 TimeOut);
}

// This class houses the public methods that we'll use from powershell
public static class StaticMethods
{

public const UInt32 CR_SUCCESS = 0;
public const UInt64 CM_REENUMERATE_SYNCHRONOUS = 1;
public const UInt64 CM_LOCATE_DEVNODE_NORMAL = 0;

public static UInt32 RescanAllDevices()
{
//only connect to local device nodes
UInt32 ResultCode = 0;
IntPtr LocalMachineInstance = IntPtr.Zero;
UInt32 DeviceInstance = 0;
UInt32 PendingTime = 30000;

ResultCode = NativeMethods.CM_Locate_DevNode_Ex(ref DeviceInstance, IntPtr.Zero, CM_LOCATE_DEVNODE_NORMAL, LocalMachineInstance);
if (CR_SUCCESS == ResultCode)
{
ResultCode = NativeMethods.CM_Reenumerate_DevNode_Ex(DeviceInstance, CM_REENUMERATE_SYNCHRONOUS, LocalMachineInstance);
ResultCode = NativeMethods.CMP_WaitNoPendingInstallEvents(PendingTime);
}
return ResultCode;
}
}
}

Step 3: Bring your wrapper class into PowerShell
Now it's time to get our wrapper into PowerShell. You have a few options here; include as an assembly, or include as source; your scenario and business needs may lend itself to one method more than the other. For this example we're going to include our wrapper as source. Obviously, including our wrapper as source code means that it's readable by anyone who opens the PowerShell script in a text editor. Including as source works particularly well for short snippets like this. Regardless of the method, you'll be using the Add-Type cmdlet, which will do all the heavy lifting for us.
I've created a new function in my PowerShell script called InitDeviceClass that'll bring our wrapper into play. You can see this function in the completed PowerShell script at the end of this post, it looks complicated, but in reality it only has two statements. The first statement defines a string variable named $SourceCode which contains the entire source for our wrapper. Just copy/paste the whole source from Visual Studio into the PowerShell ISE. The second statement does all the work, so let's take a closer look at that.
Add-Type -TypeDefinition $SourceCode
Add-Type is a new cmdlet introduce with PowerShell 2.0 that lets you define a .NET Framework class in your Windows PowerShell session. For a complete description check out the MSDN reference. The -TypeDefinition parameter is where we tell Add-Type how to find the type definition that we want to add. In this case, we're just passing it the entire C# source file, which it'll compile and make available to us. If your C# has errors in it, Add-Type will return an error letting you know that there were compile-time errors with your code.
That's it. The wrapper is now loaded and ready to use, so let's see how that works.
Step 4: Call the methods defined in your wrapper class from PowerShell
Our wrapper defines one public static method, which makes things pretty easy. All we need to do is call our init function and then the wrapper is ready to use. The syntax is exactly the same as my old [System.IO.File]::Exists() crutch.
InitDevice Class
[mattbie.examples.devices.staticmethods]::RescanAllDevices()
And there you have it. Win32, wrapped in .NET, then wrapped in PowerShell, ready to be added to your Troubleshooting Pack resolver script. Here is the complete PowerShell script. Enjoy!
# Resolver Script - This script fixes the root cause. It only runs if the Troubleshooter detects the root cause.
# Key cmdlets:
# -- get-diaginput invokes an interactions and returns the response
# -- write-diagprogress displays a progress string to the user

# Your logic to fix the root cause here


# Create a function that loads our managed code into powershell
function InitDeviceClass
{

# This is the C# source code. I've tested this in visual studio and then copy/pasted it here
# You could also load the .CS file from the disk, if you prefer

[string]$SourceCode = @"
using System.Runtime.InteropServices;
using System;

namespace mattbie.examples.devices
{
// These are the native win32 methods that we require
internal static class NativeMethods
{
[DllImport("cfgmgr32.dll", SetLastError = true, EntryPoint = "CM_Locate_DevNode_Ex", CharSet = CharSet.Auto)]
public static extern UInt32 CM_Locate_DevNode_Ex(ref UInt32 DevInst, IntPtr DeviceID, UInt64 Flags, IntPtr Machine);

[DllImport("cfgmgr32.dll", SetLastError = true, EntryPoint = "CM_Reenumerate_DevNode_Ex", CharSet = CharSet.Auto)]
public static extern UInt32 CM_Reenumerate_DevNode_Ex(UInt32 DevInst, UInt64 Flags, IntPtr Machine);

[DllImport("cfgmgr32.dll", SetLastError = true, EntryPoint = "CMP_WaitNoPendingInstallEvents", CharSet = CharSet.Auto)]
public static extern UInt32 CMP_WaitNoPendingInstallEvents(UInt32 TimeOut);
}

// This class houses the public methods that we'll use from powershell
public static class StaticMethods
{

public const UInt32 CR_SUCCESS = 0;
public const UInt64 CM_REENUMERATE_SYNCHRONOUS = 1;
public const UInt64 CM_LOCATE_DEVNODE_NORMAL = 0;

public static UInt32 RescanAllDevices()
{
//only connect to local device nodes
UInt32 ResultCode = 0;
IntPtr LocalMachineInstance = IntPtr.Zero;
UInt32 DeviceInstance = 0;
UInt32 PendingTime = 30000;

ResultCode = NativeMethods.CM_Locate_DevNode_Ex(ref DeviceInstance, IntPtr.Zero, CM_LOCATE_DEVNODE_NORMAL, LocalMachineInstance);
if (CR_SUCCESS == ResultCode)
{
ResultCode = NativeMethods.CM_Reenumerate_DevNode_Ex(DeviceInstance, CM_REENUMERATE_SYNCHRONOUS, LocalMachineInstance);
ResultCode = NativeMethods.CMP_WaitNoPendingInstallEvents(PendingTime);
}
return ResultCode;
}
}
}
"@

# use the powershell 2.0 add-type cmdlet to compile the source and make it available to our powershell session
add-type -TypeDefinition $SourceCode

}

Use PowerShell to Interact with the Windows API

In Windows PowerShell, there are three ways to interact with Windows API functions:

Use the Add-Type cmdlet to compile C# code. This is the officially documented method.
Get a reference to a private type in the .NET Framework that calls the method.
Use reflection to dynamically define a method that calls the Windows API function.
Background to using Add-Type

In the examples that follow, I use the CopyFile function in kernel32.dll as the function that Windows PowerShell will interact with. Now, you may have just asked the question, "Why would I want to call CopyFile when Windows PowerShell already has the Copy-Item cmdlet?"

That's a very good question, indeed. As it turns out though, there are certain file paths that the Windows PowerShell file provider doesn't know how to handle. In particular, it doesn't know how to interpret special device object paths, such as: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Windows\System32\calc.exe. These are the kinds of paths that you need to deal with when you interact with files that are backed up by the Volume Shadow Copy Service. For example, running the following command lists the device object paths of each volume shadow copy:

# Run this from an administrative prompt

Get-WmiObject Win32_ShadowCopy | Select-Object DeviceObject

Before diving into using Windows PowerShell to call CopyFile, it is helpful to have some background on C/C++ types vs. .NET types. According to MSDN documentation (see CopyFile function), CopyFile has the following function definition:

Image of code

So CopyFile has a return type of BOOL and three parameters?two 'T' strings and a bool. The key to interacting with Win API functions is knowing how to convert these C/C++ types to the equivalent .NET type. Fortunately, there is a site dedicated to this cause: PINVOKE.NET , which provides a treasure trove of the type of information you need to perform this task. It is worth a bookmark in your favorites.

If you search for CopyFile on PINVOKE.NET, you will find the following C# definition:

[DllImport("kernel32.dll", CharSet = CharSet.Unicode)]
static extern bool CopyFile(string lpExistingFileName, string lpNewFileName, bool bFailIfExists);

You can now see the translation of C/C++ types to .NET types:

BOOL -> bool

LPCTSTR -> string

Now, you should have the background needed to start using Windows PowerShell to interact with the CopyFile function. Let's jump into our first method of interaction, Add-Type.

Using Add-Type to call the CopyItem function

The Add-Type cmdlet is used to define .NET types that will be made available to your Windows PowerShell session. What's really cool about it is that it can compile C# code on the fly. If you view the Help for Add-Type, you'll find some good examples of how to call Windows API functions.

As an example, the following code allows me to call the CopyItem function within Windows PowerShell:

$MethodDefinition = @'

[DllImport("kernel32.dll", CharSet = CharSet.Unicode)]

public static extern bool CopyFile(string lpExistingFileName, string lpNewFileName, bool bFailIfExists);

'@

$Kernel32 = Add-Type -MemberDefinition $MethodDefinition -Name 'Kernel32' -Namespace 'Win32' -PassThru

# You may now call the CopyFile function

# Copy calc.exe to the user's desktop

$Kernel32::CopyFile("$($Env:SystemRoot)\System32\calc.exe", "$($Env:USERPROFILE)\Desktop\calc.exe", $False)

The $MethodDefinition variable simply contains the C# definition that I took from PINVOKE.NET with one minor modification: I defined the CopyFile method to be public. Methods that are added with Add-Type must be public to easily interact with them in Windows PowerShell.

I then call Add-Type and provide the C# source code, a type name, and a namespace. By specifying the type name and namespace, after calling Add-Type you can reference the new type in WindowsPowerShell with `[Win32.Kernel32]`.

Lastly, by default, Add-Type doesn't output the type definition that it creates. The -PassThru parameter tells it to output the type definition.

After calling Add-Type, you can finally call CopyFile directly within Windows PowerShell. The previous example simply copies calc.exe to the user's desktop. It's worth noting the two colons that follow the $Kernel32 variable. These indicate that you are calling a static .NET method. All static methods are called this way in Windows PowerShell.

To wrap things up, I wrote the Copy-RawItem function that wraps the CopyFile function nicely. The full script is also available in the Script Center Repository: Copy-RawItem (Add-Type Version).

Copy-RawItem Function:

function Copy-RawItem

{

<#

.SYNOPSIS

Copies a file from one location to another including files contained within DeviceObject paths.

.PARAMETER Path

Specifies the path to the file to copy.

.PARAMETER Destination

Specifies the path to the location where the item is to be copied.

.PARAMETER FailIfExists

Do not copy the file if it already exists in the specified destination.

.OUTPUTS

None or an object representing the copied item.

.EXAMPLE

Copy-RawItem '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Windows\System32\config\SAM' 'C:\temp\SAM'

#>

[CmdletBinding()]

[OutputType([System.IO.FileSystemInfo])]

Param (

[Parameter(Mandatory = $True, Position = 0)]

[ValidateNotNullOrEmpty()]

[String]

$Path,

[Parameter(Mandatory = $True, Position = 1)]

[ValidateNotNullOrEmpty()]

[String]

$Destination,

[Switch]

$FailIfExists

)

$MethodDefinition = @'

[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]

public static extern bool CopyFile(string lpExistingFileName, string lpNewFileName, bool bFailIfExists);

'@



$Kernel32 = Add-Type -MemberDefinition $MethodDefinition -Name 'Kernel32' -Namespace 'Win32' -PassThru



# Perform the copy

$CopyResult = $Kernel32::CopyFile($Path, $Destination, ([Bool] $PSBoundParameters['FailIfExists']))



if ($CopyResult -eq $False)

{

# An error occured. Display the Win32 error set by CopyFile

throw ( New-Object ComponentModel.Win32Exception )

}

else

{

Write-Output (Get-ChildItem $Destination)

}

}

The following image illustrates using the Copy-RawItem function.

Image of command output

The only code in the Copy-RawItem that warrants explanation is the last few lines where error checking occurs. The MSDN documentation states that if CopyFile returns FALSE, an error occurred. In C/C++, you would determine the cause for the error by calling GetLastError. In Windows PowerShell, you can channel the error by throwing a ComponentModel.Win32Exception object. This is what allows you to pry out the error from a Win API function without calling GetLastError.

Use PowerShell to Work with the .NET Framework Classes

Hey, Scripting Guy! AnswerHello AG, Microsoft Scripting Guy Ed Wilson here. One of the really cool things about the TechNet Script Center is the Scripting Guys Script Repository. It has come a long way since it debuted last year. Quarterly updates keep adding new features, such as the ability to select from multiple templates when you upload your script and the ability to go back in and edit the script after it is published. My favorite feature is the ability to upload a picture or even a video file to show and explain what is happening with the script. I decided to create a video to explain the script that is seen at the end of this post.

AG, before I discuss services related to device drivers, we have to first examine .NET Framework assemblies and see whether they are loaded. When it comes to seeing whether a particular .NET Framework assembly is loaded, it is easiest to use the Add-Type Windows PowerShell cmdlet, and try to load the assembly. Nothing will occur if the Add-Type cmdlet tries to load an assembly multiple times, because .NET Framework assemblies only load once. As seen in the code here, I call the Add-Type Windows PowerShell cmdlet three times, and nothing is displayed ... neither a confirmation, nor an error. This is shown here.

PS C:\> Add-Type -AssemblyName System.ServiceProcess
PS C:\> Add-Type -AssemblyName System.ServiceProcess
PS C:\> Add-Type -AssemblyName System.ServiceProcess
PS C:\>



Because no confirmation is supplied from the Add-Type cmdlet, I sometimes wonder if the assembly loaded correctly. To check this, I generally put the .NET Framework class name (together with the .NET Framework namespace the class resides in) into a pair of square brackets. This command is seen here.

PS C:\> [System.ServiceProcess.ServiceController]

IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False ServiceController System.ComponentModel....


PS C:\>



Another way to make sure that a .NET Framework class is available is to use the GetAssembly static method from the System.Reflection.Assembly .NET Framework class. The namespace is System.Reflection, and the class name is Assembly. Static methods are always available from the class. The GetAssembly static method requires an input object that is a type. One way to obtain a type is to cast a string into a type. This is seen here, where I store the type inside the $sc variable, and then pass the $sc variable to the GetAssembly method.

PS C:\> $sc = $sc = "System.ServiceProcess.ServiceController" -as [type]
PS C:\> [reflection.assembly]::GetAssembly($sc)

GAC Version Location
--- ------- --------
True v2.0.50727 C:\Windows\assembly\GAC_MSIL\System.ServiceProcess\2.0.0.0_...



The GetAssembly method returns the loaded assembly as an instance of the System.Reflection.Assembly class. This means that you can pass it to the Get-Member Windows PowerShell cmdlet and see what methods, properties and events are available. This is seen here.

PS C:\> [reflection.assembly]::GetAssembly($sc) | Get-Member





TypeName: System.Reflection.Assembly



Name MemberType Definition

---- ---------- ----------

ModuleResolve Event System.Reflection.ModuleResolveEventHandler ...

CreateInstance Method System.Object CreateInstance(string typeName...

Equals Method bool Equals(System.Object o)

GetCustomAttributes Method System.Object[] GetCustomAttributes(bool inh...

GetExportedTypes Method type[] GetExportedTypes()

GetFile Method System.IO.FileStream GetFile(string name)

GetFiles Method System.IO.FileStream[] GetFiles(), System.IO...

GetHashCode Method int GetHashCode()

GetLoadedModules Method System.Reflection.Module[] GetLoadedModules(...

GetManifestResourceInfo Method System.Reflection.ManifestResourceInfo GetMa...

GetManifestResourceNames Method string[] GetManifestResourceNames()

GetManifestResourceStream Method System.IO.Stream GetManifestResourceStream(t...

GetModule Method System.Reflection.Module GetModule(string name)

GetModules Method System.Reflection.Module[] GetModules(), Sys...

GetName Method System.Reflection.AssemblyName GetName(), Sy...

GetObjectData Method System.Void GetObjectData(System.Runtime.Ser...

GetReferencedAssemblies Method System.Reflection.AssemblyName[] GetReferenc...

GetSatelliteAssembly Method System.Reflection.Assembly GetSatelliteAssem...

GetType Method type GetType(string name), type GetType(stri...

GetTypes Method type[] GetTypes()

IsDefined Method bool IsDefined(type attributeType, bool inhe...

LoadModule Method System.Reflection.Module LoadModule(string m...

ToString Method string ToString()

CodeBase Property System.String CodeBase {get;}

EntryPoint Property System.Reflection.MethodInfo EntryPoint {get;}

EscapedCodeBase Property System.String EscapedCodeBase {get;}

Evidence Property System.Security.Policy.Evidence Evidence {get;}

FullName Property System.String FullName {get;}

GlobalAssemblyCache Property System.Boolean GlobalAssemblyCache {get;}

HostContext Property System.Int64 HostContext {get;}

ImageRuntimeVersion Property System.String ImageRuntimeVersion {get;}

Location Property System.String Location {get;}

ManifestModule Property System.Reflection.Module ManifestModule {get;}

ReflectionOnly Property System.Boolean ReflectionOnly {get;}



It is easy to see the values for the various properties that are shown by the Get-Member cmdlet. All I have to do is to pipeline the object to the Format-List cmdlet as shown here.

PS C:\> $sc = $sc = "System.ServiceProcess.ServiceController" -as [type]

PS C:\> [reflection.assembly]::GetAssembly($sc) | Format-List *





CodeBase : file:///C:/Windows/assembly/GAC_MSIL/System.ServiceProcess/2.0

.0.0__b03f5f7f11d50a3a/System.ServiceProcess.dll

EscapedCodeBase : file:///C:/Windows/assembly/GAC_MSIL/System.ServiceProcess/2.0

.0.0__b03f5f7f11d50a3a/System.ServiceProcess.dll

FullName : System.ServiceProcess, Version=2.0.0.0, Culture=neutral, Publi

cKeyToken=b03f5f7f11d50a3a

EntryPoint :

Evidence : {<System.Security.Policy.Zone version="1">

<Zone>MyComputer</Zone>

</System.Security.Policy.Zone>

, <System.Security.Policy.Url version="1">

<Url>file:///C:/Windows/assembly/GAC_MSIL/System.ServiceProces

s/2.0.0.0__b03f5f7f11d50a3a/System.ServiceProcess.dll</Url>

</System.Security.Policy.Url>

, <System.Security.Policy.GacInstalled version="1"/>

, <StrongName version="1"

Key="002400000480000094000000060200000024000052534131000400000

100010007D1FA57C4AED9F0A32E84AA0FAEFD0DE9E8FD6AEC8F87FB03766C8

34C99921EB23BE79AD9D5DCC1DD9AD236132102900B723CF980957FC4E1771

08FC607774F29E8320E92EA05ECE4E821C0A5EFE8F1645C4C0C93C1AB99285

D622CAA652C1DFAD63D745D6F2DE5F17E5EAF0FC4963D261C8A12436518206

DC093344D5AD293"

Name="System.ServiceProcess"

Version="2.0.0.0"/>

...}

ManifestModule : System.ServiceProcess.dll

ReflectionOnly : False

Location : C:\Windows\assembly\GAC_MSIL\System.ServiceProcess\2.0.0.0__b0

3f5f7f11d50a3a\System.ServiceProcess.dll

ImageRuntimeVersion : v2.0.50727

GlobalAssemblyCache : True

HostContext : 0







PS C:\>



Most of the time I do not actually need all this information. However, it is useful to see where the information is obtained and to have an idea of what information is available. I decided to use this information, to write the Get-DeviceDriverServices.ps1 script seen here.

Get-DeviceDriverServices.ps1

#Requires -version 2.0
Function Get-DeviceDriverService
{
Param(
[string]$computer = "localhost"
)
Add-Type -AssemblyName System.ServiceProcess
[System.ServiceProcess.ServiceController]::GetDevices($computer)
}

PowerShell の Add-Type と [Reflection.Assembly]

Add-Type の方が、[reflection.assembly]::LoadWithPartialName() よりいいよ!やったね。と書こうと思ったのです。 はじめは。

大体の場合は、 Add-Type はイイ感じに動作します。が、せっかくAdd-Type を作るときに Microsoft はちょっと、それは。と思うような動作も許してしまっています。 なので、結局一概にはいえないな、と思うのでちょっと記事にしておきます。

[Reflection.Assembly]
Windows は、 .Net を利用することで快適に使えます。ただ.Net は膨大な規模なのですけど、PowerShell のデフォルトでは コアになる部分しか読み込まれていません。

PowerShell で デフォルトで参照されていない.Net を利用したい。わけで、.Netアセンブリをメモリに読み込む機会も多々あります。*1

たとえば、.Net 4.5 で追加された HttpClient を利用しようとすると

Unable to find type [System.Net.Http.HttpClient]. Make sure that the assembly that contains this type is loaded.
At line:1 char:1
+ [System.Net.Http.HttpClient]
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.Http.HttpClient:TypeName) [], RuntimeException
+ FullyQualifiedErrorId : TypeNotFound
PowerShell 1.0 では、明示的に参照するために [System.Reflection.Assembly] のStaticメソッドで読み込みをしました。

あ、System は省略可能なので、 [Reflection.Assembly] と書けます。

Load メソッド
まずは Load() メソッドでしょうか。え、使わない?ソウデスネ。Fullname を求めるので使ってられません。そんなの覚えてられるか。書きたくない!

[Reflection.Assembly]::Load("System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")
LoadFrom メソッド
それなら LoadFrom() って? C# のように参照に追加してあとはさくっと名前で参照とかできないのでツラいですね。

[Reflection.Assembly]::LoadFrom("C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5.1\System.Net.Http.dll")
LoadWithPartialName() メソッド
で、良く出てきているのがこれです。

[Reflection.Assembly]::LoadWithPartialName("System.Net.Http")
気分的には こっちが見慣れた フルネーム ですし、うまく動いていますね。しかし、Stringなのでタブ補完効かないしめんどくさい。

[System.Net.Http.HttpClient]
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False HttpClient System.Net.Http.HttpMessageInvoker
Add-Type
PowerShell 2.0 で、 PowerShell は改善策を打ってきました。それが、Add-Type Cmdlet です。

たとえば、先ほどの

[Reflection.Assembly]::LoadWithPartialName("System.Net.Http")
は、Add-Type でこう書けます。

Add-Type -AssemblyName System.Net.Http
これなら、タブ補完も効くしいい感じですね!やった!だいたいの場合は。

問題は、Partial Name で判別ができない場合です。

複数バージョンのアセンブリがインストールされている場合が問題

しかし、 Add-Type は、これとは別の内部テーブルを使っているようで、 [Reflection.Assembly]::LoadWithPartialName()とは違う結果がもたらされます。

もし 複数のアセンブリが異なるバージョンでインストールされている場合、 Add-Type -AssembryName は判別する方法を持ちません。で、だいたいの場合は古いバージョンが参照されて、新しいバージョンを参照しているスクリプトは失敗します。

Add-Type の失敗例
例えば SQLServer,SMO は、厄介です。

v10.0.0.0 と v11.0.0.0 などがありますが、Add-Type で最新が参照されるでしょーっと気にせずに追加しようとすると....

Add-Type -AssemblyName Microsoft.SqlServer.SMO
PowerShellエンジンは 古いバージョンを読み込むどころか、アセンブリの読み込みに失敗します。v9.0.242.0 って、やだー。ないよそんなのれがしー。

Add-Type : Could not load file or assembly 'Microsoft.SqlServer.Smo, Version=9.0.242.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.
At line:1 char:1
+ Add-Type -AssemblyName Microsoft.SqlServer.SMO
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Add-Type], FileNotFoundException
+ FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.AddTypeCommand
Add-Type ではフルネームが必要になる場合がある
この場合、バージョンを的確に指定する、つまりフルネームならいけます。

例えば私の環境*2の場合は、v11.0.0.0 を指定すればokです。

Add-Type -AssemblyName "Microsoft.SqlServer.SMO, Version=11.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91"
しかし、もう一回言いますがフルネームはない。

[Reflection.Assembly]::LoadWithPartialName() なら大丈夫

Microsoft.SqlServer.Smo だけで読み込んで欲しいじゃないですか。で、これならok。

[Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo")
はい、ok。

GAC Version Location
--- ------- --------
True v2.0.50727 C:\Windows\assembly\GAC_MSIL\Microsoft.SqlServer.SMO\11.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.SMO.dll
ただし、この場合は どのバージョンが読み込まれるかは保証できません、が、とりあえず読んでくれる。やった。Add-Typeどの〜。

PowerShell の Type Conversion
この問題には、 PowerShell の型変換システムが絡んでいます。

[Reflection.Assembly]::LoadWithPartialName() メソッドは裏でうまくアセンブリを判別し、またPowerShellの型変換アルゴリズムを使ってくれています。素晴らしい。*3

PowerShell の10ある型変換アルゴリズムはここを見てください。

Understanding PowerShell's Type Conversion Magic
Direct assignment. If your input is directly assignable, simply cast your input to that type.
Language-based conversion. These language-based conversions are done when the target type is void, Boolean, String, Array, Hashtable, PSReference (i.e.: [ref]), XmlDocument (i.e.: [xml]). Delegate (to support ScriptBlock to Delegate conversions), and Enum.
Parse conversion. If the target type defines a Parse() method that takes that input, use that.
Static Create conversion. If the target type defines a static ::Create() method that takes that input, use that.
Constructor conversion. If the target type defines a constructor that takes your input, use that.
Cast conversion. If the target type defines a implicit or explicit cast operator from the source type, use that. If the source type defines an implicit or explicit cast operator to the target type, use that.
IConvertible conversion. If the source type defines an IConvertible implementation that knows how to convert to the target type, use that.
IDictionary conversion. If the source type is an IDictionary (i.e.: Hashtable), try to create an instance of the destination type using its default constructor, and then use the names and values in the IDictionary to set properties on the source object.
PSObject property conversion. If the source type is a PSObject, try to create an instance of the destination type using its default constructor, and then use the property names and values in the PSObject to set properties on the source object. . If a name maps to a method instead of a property, invoke that method with the value as its argument.
TypeConverter conversion. If there is a registered TypeConverter or PSTypeConverter that can handle the conversion, do that. You can register a TypeConverter through a types.ps1xml file (see: $pshome\Types.ps1xml), or through Update-TypeData.
しかし、Add-Type は この型変換を使わず、内部に変換テーブルを持っているようです。で、失敗すると。

しかも、Add-Type があるためか、 [Reflection.Assembly]::LoadWithPartialName() は、deprecated 予定とか PowerShell Team も言ってますし、ほげー。*4

結論
もし、 Add-Type でアセンブリが読める + 読まれたバージョンで問題がないなら

Add-Type -AssemblyName System.Net.Http
もし、Add-Type の内部テーブルにアセンブリがなく読み込めない、バージョンがわからないけどとりあえずどれが読まれてもいいなら、

[Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Smo")
もし、アセンブリのバージョンを指定したい、または将来 上の2つが使えなくなった場合は、

Add-Type -AssemblyName "Microsoft.SqlServer.SMO, Version=11.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91"
がいいでしょう。

まとめ
Add-Type かわいいよ。もっと賢くなるともっと好き。

*1:PowerShell 3.0 以上で明示的に読む必要がいくつかでなくなっており便利になっています。

*2:Windows 8.1 Pro / Visual Studio 2013

*3:他のCmdletでもPowerShellの型変換はこれに従います

*4:Assembly.LoadWithPartialName メソッド (String)

2015年2月26日木曜日

Windows版psqlで文字化けする

インストーラのインストールだけだと、
コマンドラインの日本語がばける

Windows XP/Visuta/7
PostgreSQL 9.0 以降

この事象は既知の問題のようで、
http://ftp.gnome.org/pub/gnome/binaries/win64/dependencies/gettext-runtime_0.18.1.1-2_win64.zip
ここから、アーカイブをダウンロードし
bin 配下の libintl-8.dll を
C:\Program Files\PostgreSQL\9.1\bin\libintl.dll

に置き換えます
ファイル名は「libintl.dll」です
(元のファイルは、「libintl.dll.org」と、名前を 変えておいておいた方がよいでしょう。)
置き換えるには、
一度、PostgreSQL のサービスを停止した上で、
置き換えてください。
(管理者権限が必要です)
置き換えたあとは、PostgreSQLを再起動です。

2015年2月25日水曜日

upload file asynchronously with Jquery

To upload file asynchronously with Jquery use below steps:

step 1 In your project open Nuget manager and add package (jquery fileupload(only you need to write it in search box it will come up and install it.)) URL: https://github.com/blueimp/jQuery-File-Upload

step 2 Add below scripts in the HTML files, which are already added to the project by running above package:

jquery.ui.widget.js

jquery.iframe-transport.js

jquery.fileupload.js

step 3 Write file upload control as per below code:

<input id="upload" name="upload" type="file" />

step 4 write a js method as uploadFile as below:

 function uploadFile(element) {                $(element).fileupload({                    dataType: 'json',                  url: '../DocumentUpload/upload',                  autoUpload: true,                  add: function (e, data) {                               // write code for implementing, while selecting a file.                     // data represents the file data.                     //below code triggers the action in mvc controller                    data.formData =                                      {                                       files: data.files[0]                                      };                    data.submit();                  },                  done: function (e, data) {                               // after file uploaded                  },                  progress: function (e, data) {                       // progress                  },                  fail: function (e, data) {                       //fail operation                  },                  stop: function () {                      code for cancel operation                  }              });            };

step 5 In ready function call element file upload to initiate the process as per below:

$(document).ready(function()  {      uploadFile($('#upload'));    });

step 6 Write MVC controller and Action as per below:

public class DocumentUploadController : Co  ntroller      {                   [System.Web.Mvc.HttpPost]          public JsonResult upload(ICollection<HttpPostedFileBase> files)          {              bool result = false;                if (files != null || files.Count > 0)              {                  try                  {                      foreach (HttpPostedFileBase file in files)                      {                          if (file.ContentLength == 0)                              throw new Exception("Zero length file!");                                                 else                               //code for saving a file                        }                  }                  catch (Exception)                  {                      result = false;                  }              }                  return new JsonResult()                  {                      Data=result                  };              }        }