StackOverflow PowerShell API Clien Sample

Stack Overflow you guys are awesome, thank you for being open!

First of all there is StackExchange Data Explorer which allow you to run SQL queries like this over Stack Overflows databases which is awesome!

Second stuff is StackExchange API which allows you to retrieve some interesting data via REST API.

Here is example of such request and its response:

https://api.stackexchange.com/2.2/tags/synonyms?site=stackoverflow&pagesize=5

{
  "items": [
    {
      "creation_date": 1427427681,
      "applied_count": 0,
      "to_tag": "css",
      "from_tag": "font-weight"
    },
    {
      "creation_date": 1428794822,
      "last_applied_date": 1429012623,
      "applied_count": 6,
      "to_tag": "javascript",
      "from_tag": "javascript-library"
    },
    {
      "creation_date": 1364923625,
      "last_applied_date": 1429037717,
      "applied_count": 1,
      "to_tag": "sql-like",
      "from_tag": "like-operator"
    },
    {
      "creation_date": 1428596908,
      "last_applied_date": 1429223402,
      "applied_count": 37,
      "to_tag": "swift",
      "from_tag": "swift1.2"
    },
    {
      "creation_date": 1285959378,
      "applied_count": 0,
      "to_tag": "each",
      "from_tag": ".each"
    }
  ],
  "has_more": true,
  "quota_max": 300,
  "quota_remaining": 197
}

To retrieve full list of synonyms (which can be later used in stuffs like ElasticSearch) you could use following script:

$body = @{
    site = 'stackoverflow'
    pagesize = 100
    page = 1
}

if($key) {
    $body['key'] = $key
}

$items = @()

$body['filter'] = 'total'
$response = Invoke-RestMethod -Uri "https://api.stackexchange.com/2.2/tags/synonyms" -Body $body
$pages = $response.total / $body['pagesize']
$body.Remove('filter')

do {
    Write-Progress -Activity 'Retrieving page' -Status $body['page'] -PercentComplete ( [Math]::Min($body['page'] / $pages * 100, 100) )
    $response = Invoke-RestMethod -Uri "https://api.stackexchange.com/2.2/tags/synonyms" -Body $body
    $items += $response.items
    $body['page'] += 1
} while($response.has_more)

$items | select -First 10 | select @{n='From';e={ $_.from_tag.Replace('-', ' ') }}, @{n='To';e={ $_.to_tag.Replace('-', ' ') }}

Which will produce output like this:

From               To
----               --
font weight        css
javascript library javascript
like operator      sql like
swift1.2           swift
.each              each
column store index columnstore
android ui         android
uber               uber api
elixir lang        elixir
css box model      css

While requesting API you should define site data from which you want to retrieve, to get list of available sites you can use something like this:

$body = @{
    pagesize = 100
    page = 1
}

if($key) {
    $body['key'] = $key
}

$items = @()

do {
    $response = Invoke-RestMethod -Uri "https://api.stackexchange.com/2.2/sites" -Body $body
    $items += $response.items
    $body['page'] += 1
} while($response.has_more)

$items | where site_type -eq 'main_site' | where site_state -eq 'normal' | select name, api_site_parameter, audience | ft -AutoSize

Here is list of "normal" main (not meta) sites:

name                           api_site_parameter
----                           ------------------
Stack Overflow                 stackoverflow
Server Fault                   serverfault
Super User                     superuser
Meta Stack Exchange            meta
Web Applications               webapps
Arqade                         gaming
Webmasters                     webmasters
Seasoned Advice                cooking
Game Development               gamedev
Photography                    photo
Cross Validated                stats
Mathematics                    math
Home Improvement               diy
Geographic Information Systems gis
TeX - LaTeX                    tex
Ask Ubuntu                     askubuntu
Personal Finance & Money   money
English Language & Usage   english
Stack Apps                     stackapps
User Experience                ux
Unix & Linux               unix
WordPress Development          wordpress
Theoretical Computer Science   cstheory
Ask Different                  apple
Role-playing Games             rpg
Bicycles                       bicycles
Programmers                    programmers
Electrical Engineering         electronics
Android Enthusiasts            android
Physics                        physics
Information Security           security
Graphic Design                 graphicdesign
Database Administrators        dba
Science Fiction & Fantasy  scifi
Skeptics                       skeptics
Drupal Answers                 drupal
SharePoint                     sharepoint
Mi Yodeya                      judaism
Travel                         travel
Christianity                   christianity
Movies & TV                movies
Mathematica                    mathematica
Academia                       academia
The Workplace                  workplace
Salesforce                     salesforce
ExpressionEngine® Answers expressionengine
MathOverflow                   mathoverflow.net

To be able to make more requests you should register your app.

Unfortunatelly there is not API for careers but there is feed that contains interesting information, here is few samples what can be done:

Firs of all we need retrieve feed itself and process it:

$xml = [xml](Invoke-WebRequest http://careers.stackoverflow.com/jobs/feed)

$items = @()
foreach($item in $xml.rss.channel.item) {
    if($item.category) {
        $parts = $item.title.Substring($item.title.LastIndexOf('(')).Trim(@(' ', '(', ')')).Split(',')

        $city = $parts | select -First 1
        $region = $parts | select -Last 1

        foreach($category in $item.category) {
            $el = New-Object psobject
            $el | Add-Member NoteProperty 'City' $city.Trim()
            $el | Add-Member NoteProperty 'Region' $region.Trim()
            $el | Add-Member NoteProperty 'Category' $category
            $items += $el
        }
    }
}

Now we can get some aggregated data

Top 10 vacancies by region

$items | Group-Object Region | select Name, Count | Sort-Object Count -Descending | select -First 10 | ft -AutoSize
Name          Count
----          -----
UK              512
CA              479
Deutschland     373
allows remote   291
NY              290
Netherlands     195
Germany         190
WA              123
Canada          114
Sweden           95

Top 10 vacancies by category

$items | Group-Object Category | select Name, Count | Sort-Object Count -Descending | select -First 10 | ft -AutoSize
Name       Count
----       -----
java         246
javascript   223
python       128
c#           108
php          107
c++          103
sql           84
linux         79
mysql         73
css           70

Top 10 vacancies by both region and category

$items | Group-Object Region, Category | Sort-Object Count -Descending | select @{n='Region';e={ $_.Values[0] }}, @{n='Category';e={ $_.Values[1] }}, Count | select -First 10 | ft -AutoSize
Region        Category   Count
------        --------   -----
UK            java          34
UK            javascript    33
CA            java          27
CA            javascript    26
NY            java          23
Deutschland   java          23
allows remote javascript    22
CA            python        22
UK            c#            20
UK            php           20

Google Pie chart can be made from PowerShell with that piece of code:

$chart = $items | Group-Object Region | select Name, Count | Sort-Object Count -Descending | select -First 5
$chd = 't:' + (($chart | select -ExpandProperty Count) -join ',')
$chl=($chart | select @{n='Label';e={ $_.Name + ' - ' + $_.Count }} | select -ExpandProperty Label) -join '|'
"https://chart.googleapis.com/chart?cht=p&chs=600x200&chd=$chd&chl=$chl"