https with ngrok

While ngrok can be used to expose your local development application to the internet for testing,
the free version of ngrok does not support End-to-End TLS Tunnels aka https, but it does support forwarding of https to http

Note: Transport Layer Security (TLS) is the successor to SSL. TLS 1.0 was defined in RFC 2246 in January 1999.  Hypertext Transfer Protocol Secure (HTTPS), or “HTTP Secure,” is an application-specific implementation that is a combination of the Hypertext Transfer Protocol (HTTP) with the SSL/TLS.
reference: is-it-ssl-tls-or-https

For Single Sing On solutions, such as SAML, you application is required to be hosted over https

You might think you could just bind to the ssl port 443

> ngrok http myawesomeapp.local:443

Session Status                online
Session Expires               7 hours, 4 minutes
Version                       2.2.8
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://8316dcb8.ngrok.io -> myawesomeapp.local:443
Forwarding                    https://8316dcb8.ngrok.io -> myawesomeapp.local:443  

But, accessing https://8316dcb8.ngrok.io or http://8316dcb8.ngrok.io results in the browser showing

Bad Request
Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.

So that is why you need the paid version of ngrok for End-to-End TLS Tunnels.

One solution is to pay for the Pro version of ngrok, which also gets you access to more features such as Whitelabel domains, Reserved TCP addresses, End-to-End TLS Tunnels, and more resources.

However, if your using PHP Symfony or the PHP OneLogin SAML library,
another solution is to modify your application to indicate that the request is actually https.

Start up ngrok

> ngrok http myawesomeapp.local:80

Session Status                online
Session Expires               7 hours, 4 minutes
Version                       2.2.8
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://8316dcb8.ngrok.io -> myawesomeapp.local:80
Forwarding                    https://8316dcb8.ngrok.io -> myawesomeapp.local:80  

If you view https://8316dcb8.ngrok.io, you will see your app, but you app will think its running on port 80 and http

Note: While this is PHP focused, the concept should apply to other applications.

To make your app think it's on https, in the bootstrap or initialization of your application, add:

// If the host url has ngrok.io, and the header X-Forwarded-Proto set by ngrok is https, then also set https on and the port to 443
if (strpos($_SERVER['HTTP_HOST'], 'ngrok.io') !== false && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
{
  $_SERVER['HTTPS'] = 'on';
  $_SERVER['SERVER_PORT'] = 443;
}

Note: For Symfony, you can place this code within the boostrap app_dev.php, above the creation of $request

> vi app_dev.php

// place here

$request = Request::createFromGlobals();
$response = $kernel->handle($request);


This will allow any following code which checks for https
such as OneLogins Utils::isHTTPS()
or Symfony Request->isSecure()
to return true when accessing the ngrok url over https

Note: This does not test or affect your applications ssl certs, as the browser thinks the request is from 8316dcb8.ngrok.io,  And if you are using ngrok daily for development or testing, it is fairly inexpensive to upgrade to the tls version which is a  simpler and better solution

Hopefully this helps you test Single Sing On solutions, such as SAML.

-End of Document-

Thanks for reading

streamWriter - play and record internet radio stations

streamWriter is a free application for windows that plays and records music broadcasted by internet radio stations.  The internet radio stations listed in streamWriter may stream songs you know, definitely songs you don’t know, or interesting re-mixes of your favorites.  The internet radio stations are free in the sense that FM/AM is free.

Note some radio stations may contain commercials, or be in other languages than your own.

streamWriter features

• Record as many streams as you want at the same time (MP3/AAC)
• Automatically record a wishlist's song when it's playing on a stream
• Player for streams and recorded files
• Track splitting with silence detection
• Function for manual cutting of saved titles
• Tracks are named by a given pattern
• Short songs (ads) can be skipped
• Writing of tags to recorded files
• Script-based postprocessing
• Applying of effects to recorded songs (SoX)
• Integration of different audio encoders
• Scheduled recordings
• Stream browser
• Multilingual
• Can be installed or used in portable mode
• ...and much more!

streamWriter Website Help Donate

This post will cover the installation, recommended settings, and general usage of streamWriter.

Installation

1) Downloading and installing the executable or portable version

Run streamWriter

You will be greeted with some info and a settings wizard.  

You can change all settings later.

2) About, License, Thank you dialog

It’s unusual but kinda nice that you see the about dialog first.

Click Close

3) Choose your language

Click Next

4) Save settings location

Choose either option.  

The second option to Save data to application path is recommended to make the app portable.

Click Next

5) Choose to search for updates

Leave enabled is fine.

You will have to download and install new versions, which is safer than auto upgrades.

Click Next

6) Select recording folder

Choose the folder (directory) where you want streamed recordings to be saved.  

The default under Music is fine.

Click Next

7) Some miscellaneous settings

The defaults are fine.

Click Finish

8) Introduction

Some nice info, but you will probably forget all the words,

so continue on for more screenshots and info

Click Close

9) You now see the full app

Choose a radio stream

1) Filter your desired music by

• Genre - Categorizes the type of music i.e. Country, Rock, etc.  
• There are lots of Genres, and many overlap.

It’s more of a guideline than a rule.  

So some trial and error will be required to find what you like.

• Kbps - Determines the quality of the songs being streamed. The higher the better quality, and bigger file size. Note, some music players may not play mp3s with high Kbps (also known as Bitrate).

Choose >= 192 should be a good mix of quality, size, and compatibility

• Type - The file format of the radio stream.

Choose MP3 for greater compatibility.

2) Choose a radio stream

Double click on a radio stream, listed on the bottom right side.  

The radio stream will start.  

By default, you will not hear anything as the radio stream will only record.

3) Listen to the radio stream

Click the Play button to listen while recording.

Recommended Settings

1) Go to settings...

Click File -> Settings..

2) General Settings

Enable

• Remember streams that were recording on exit
• Remember stream that was playing on exit

So when you restart the app, it continues playing the same radio stream.

3) Recordings

Enable

• Only save whole songs
• Remove saved titles from wishlist (you can add songs to wishlists, ignore lists)
• Overwrite existing file if newer file is larger
• Discard new file if existing one is larger

The last two options should increase the odds of recording a better quality song

4) Filenames

You can change the location where songs are recorded too ie saved too

For greater organization, change the Patterns for filenames of recorded tracks to include the artist as a folder

• Original:
• %streamname%\%artist% - %title%
• Recommend folders by artist:
• %streamname%\%artist%\%artist% - %title%

The defaults on Filenames Advanced and Cut songs are fine.

Note some radio streams do have commercials, of course.  

The defaults on Filenames Advanced and Cut songs seem to be ok, but you can always try changing them if needed.

5) Addons

Click and download the addons for

• Support encoding of MP3 using LAME (an encoder)
• Support reading/writing of tags using AudioGenie (a tag writer)

Writing tags to your MP3s allows audio players to display more information about the song, such as Artist, Title, Album, if known.

6) Postprocessing

Post processing is what happens to the song recording after being saved to a file.

• Select MP3 as the format to convert recorded files to,

Which ensures you are only dealing with MP3s, for greater compatibility with audio players.

• Enabled Write tags to recorded songs.

Writing tags to your MP3s allows audio players to display more information about the song, such as Artist, Title, Album, if known.

Note that these settings will only apply to new radio streams.  So you will want to remove any of your current radio streams and re-add them.  Or choose and set the same settings by right clicking on the radio stream and selecting Settings..

The defaults on the rest of the Settings: Bandwidth, Community, Hotkeys, Advanced, and General are fine.

Manage radio stream

1) While a radio stream is in the play list, and selected, you can use the toolbar button, or right click and apply some options.

If there is a song currently playing which you really do not like,  

• Choose Add title to global ignorelist and you will not hear it again.

Each stream can have it’s own recording settings, under Settings..

2) Saved songs

Provides a basic music player.

You may find the music player foobar or the video/audio player videolan useful, but the default windows player is fine too.

3) Title search

Searching for a know title can help you find a radio stream you may like.

You can also right click and add the title or artist to your wish list.

4) Play or Record or Both or Neither

Well, that’s it.

Hopefully you a good start on listening and recording internet radio streams.

streamWriter  Website Help Donate

-End of Document-

Thanks for reading

ngrok - expose a local server to the internet

• Whats is ngrok?

ngrok provides introspected tunnels to localhost.
ngrok is a reverse proxy that creates a secure tunnel from a public endpoint to a locally running web service.

Which translates to
"I want to expose a local server to the internet"
http://anyonecanaccess -> http://localhost

• Why would you want to use ngrok?

ngrok can be necessary for development and testing external services, such as Single Sign On, SAML, external APIs, Cloud solutions, etc
While you could pay for a domain and forward that to your development environment, that tends to be cost prohibitive and not overly practical given the nature of development and testing vms, containers, services, etc.  Especially in a company environment where you do not have access to the routers/firewalls.

You can also use ngrok to quickly demo a local feature before deploying.

• How to use ngrok

To use ngrok, download and extract it to a directory
https://ngrok.com/download
C:/dev/ngrok

From a command prompt,
such as microsofts cmd or a more feature rich ConsoleZ

> ngrok http 80

Session Status         online
Session Expires        7 hours, 4 minutes
Version                2.2.8
Region                 United States (us)
Web Interface          http://127.0.0.1:4040
Forwarding             http://24f98db7.ngrok.io -> localhost:80
Forwarding             https://24f98db7.ngrok.io -> localhost:80  

Assuming you already have a web based application running on port 80 ie
http://localhost
you can access your local development environment via
http://24f98db7.ngrok.io
The '24f98db7' part is randomly generated every time you start an ngrok instance.

The free version doesn't require an account and allows for basic http/tcp tunnels, which tends to be sufficient for development.  But the publicly exposed url is limited to 8 hours.  You can also create a free account for more resources if needed.

The paid versions of ngrok allow for End-to-End TLS Tunnels, Reserved domains, Reserved TCP addresses, more connections / minute, more tunnels/ngrok process, etc.

If your local development application runs on a custom local domain name such as myawesomeapp.local, you can use that with ngrok instead of localhost

> ngrok http myawesomeapp.local:80

Session Status                online
Session Expires               7 hours, 4 minutes
Version                       2.2.8  
Region                        United States (us)      
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://8316dcb8.ngrok.io -> myawesomeapp.local:80
Forwarding                    https://8316dcb8.ngrok.io -> myawesomeapp.local:80      

Note: Unlike .local, .test, and .example, .dev is not on a list of specially protected names.
in 2017, Chrome forces connections to all domains ending in .dev (as well as .foo) to use HTTPS
reference: theregister goole dev network

You can also add basic password protection to your exposed application

> ngrok http -auth "user:password" myawesomeapp.local:80

To stop ngrok just press ctrl + c

ngrok away

-End of Document-

Thanks for reading

HTTP Security Headers

When you view a website page or make a request against an application api, HTTP headers allow the client and the server to pass additional information with the request, page content, or the response.
Typical HTTP headers sent are Host, Accept-Language, etc, while typical HTTP headers received are Content-Type, Server, Content-Security-Policy, etc

You can view HTTP Headers using the browsers developer tools in Chrome or Firefox.

HTTP Security Headers are a subset of HTTP headers which can help increase the security of your web application and website. In many cases they are easy to implement and only require a slight web server configuration or application change. 

For the additional security to be realized, the browser must support the HTTP Security Headers, which most modern browsers do.  Can I Use and Mozilla are both good sites to see what features browsers support.

HTTP Security Headers to implement:

Access-Control-Allow-Origin Mozilla
Indicates whether the response can be shared with requesting code from the given origin.

Possible values:
*
- allow requesting code from any origin to access the resource

https://example.com/
- one domain

Recommended:
*
To allow Cross-origin resource sharing


X-XSS-Protection Mozilla
Stops pages from loading when they detect reflected cross-site scripting (XSS) attacks
Non standard, deprecated by Content-Security-Policy, but maybe useful for older browsers

Possible values:
0
- disables XSS filtering; never use!
1
- enable XSS filtering, sanitize, usually default in browsers
1; mode=block
- enable, block rendering of page

Recommended:
1
To silently filter XSS and not server as a simple test bed for XSS attacks


X-Content-Type-Options Mozilla
Prevents browser from MIME-type sniffing a response away from the declared ie trust the web server.

Possible values:
nosniff
- prevents browser from MIME-type sniffing a response away from the declared content-type.

Recommended:
nosniff

Good additional information on attacks mitigated (MIME Confusion Attack, Unauthorized Hotlinking) stackoverflow


X-Frame-Options Mozilla
Indicate whether or not a browser should be allowed to render a page in a <frame>, <iframe> or <object>
Non standard, deprecated by Content-Security-Policy, but useful for older browsers

Possible values:
deny
- no framessameorigin
- allow frame from origin
allow-from https://example.com/
- one domain

Recommended:
If you have no iframes, then deny
If you want your content iframed in by multiple websites, then do not send this header


Strict-Transport-Security Key CDN
Restricts web browsers to access web servers solely over HTTPS; header only has effect when requested over HTTPS

Possible values:
max-age
- defines the time in seconds for which the web server should only deliver through HTTPS.
includeSubDomains
- optional, apply to subdomains
preload
- optional, the site owner can submit their website to the preload list which is a list of sites hardcoded into Chrome as being HTTPS only; Additional details serverfault

Recommended:
If your site and all resources (images, javascript, css, etc) are available over HTTPS, as they should be, then enable; If you have mixed content, HTTP and HTTPS, then do not use this header
Given that browser will cache this header for your site, to test implementation, bump max-age up incrementally over time eg 300s=5min, 86400s=1day, 63072000s=2years required for preload
Strict-Transport-Security: max-age=300; includeSubDomains

Good additional information on why to use it stackoverflow, and what to be careful of stackoverflow


Referrer-Policy ScottHelme
Determine what information about the origin the user came from is sent to the destination site
Possible values:
no-referrer
- not referrer sent
no-referrer-when-downgrade
- do not send the referrer header when navigating from HTTPS to HTTP
same-origin
- only set the referrer header on requests to the same origin
origin
- set the referrer header to the origin, stripping any path information
strict-origin
- same as origin, but do not send HTTPS request on HTTP
origin-when-cross-origin
- send the full URL to requests to the same origin but only send the origin when requests are cross-origin
strict-origin-when-cross-origin
- same as origin-when-cross-origin, but do not send when navigating from HTTPS to HTTP
unsafe-url
- always send the referrer; do not use!

Recommended:
no-referrer-when-downgrade
To prevent any HTTPS info (referrer url) being sent over HTTP


Content-Security-Policy Key CDN Mozilla
Content-Security-Policy (CSP) is an added layer of security that helps to detect and mitigate certain types of attacks, including Cross Site Scripting (XSS) and data injection attacks.  It replaces several of the above X- headers, but support depends on browser and browser versions, so you should still the above headers.

CSP makes it possible for server administrators to reduce or eliminate the vectors by which XSS can occur by specifying the domains that the browser should consider to be valid sources of executable scripts. A CSP compatible browser will then only execute scripts loaded in source files received from those white listed domains, ignoring all other script.

Note: While searching for CSP policy values, remember CSP version 2 is currently defined, and CSP version 3 is in the works.  Across the versions, policy values have been added and removed, thus support depends on browsers and browser's versions. 

An example Content-Security-Policy:
Content-Security-Policy "base-uri 'self'; object-src 'none'"

Possible policy keys:
default-src
- default policy for all resources type that are not defined (fallback)
script-src
- which scripts the protected resource can execute
object-src
- from where the protected resource can load plugins (flash, java, etc)
style-src
- which styles (CSS) the user applies to the protected resource
img-src
- from where the protected resource can load images
media-src
- from where the protected resource can load video and audio
frame-src
- from where the protected resource can embed frames
frame-ancestors
- valid parents that may embed a page using <frame>, <iframe>, <object>, <embed>, or <applet>
font-src
- from where the protected resource can load fonts
connect-src
- which URIs the protected resource can load using script interfaces
form-action
- which URIs can be used as the action of HTML form elements
script-nonce
- script execution by requiring the presence of the specified nonce (cryptographic number used once) on script elements
report-uri
- Specifies a URI to which the user agent sends reports about policy violation

Deprecated keys (you may run across in searches, but don't use)
reflected-xss
- instructs a user agent to activate or deactivate any heuristics used to filter or block reflected cross-site scripting attacks, equivalent to the effects of the non-standard X-XSS-Protection header
referrer
- determine what information about the origin the user came from is sent to the destination site

Possible values: blobfolio
*
– wildcard, i.e. anything goes
'none'
– load no resources
'self'
– same-origin is OK
data:
– data-URI, such as a base64-encoded image
https:
– any resource over HTTPS
domain.com, *.domain.com, https://domain.com
– domain.com (any protocol), all subdomains of domain.com (any protocol), domain.com (SSL) respectively

For scripts and stylesheets specifically, there are a few additional magic values:
'unsafe-eval'
– allow scripts to run eval().
'unsafe-inline'
– allow all inline scripts and/or styles.
'nonce-XXX'
– allow inline or linked assets with the nonce stackoverflow

For more options and information refer to Mozilla

Recommended values:
A minimal CSP which should not break stuff is:
Content-Security-Policy base-uri 'self'; object-src 'none'
This ensures your base uri is not change via html injection, and that your site does not allow flash or applets
Adding CSP does require you know your application or websites resources, which may be non trivial.
Adding CSP can break you application or website, by preventing resources from loading.
For example,  object-src 'none' disable embedded pdfs in Chrome.
Testing, as always, is important.  To facilitate testing, consider adding one policy at a time.

Example Content-Security-Policy of a few domains:
google.com
header not set
amazon.com
header not set
aws.amazon.com
header not set
mail.google.com
script-src 'unsafe-inline' 'unsafe-eval' https: http:;object-src 'none';base-uri 'self';report-uri /cspreport
securityheaders.com
default-src 'self'; script-src 'self' cdnjs.cloudflare.com; img-src 'self'; style-src 'self' 'unsafe-inline' fonts.googleapis.com cdnjs.cloudflare.com; font-src 'self' fonts.gstatic.com cdnjs.cloudflare.com; form-action 'self'; report-uri https://scotthelme.report-uri.com/r/default/csp/enforce
msn.com
default-src 'self' data: 'unsafe-inline' 'unsafe-eval' https: blob:; media-src 'self' https: blob:; worker-src 'self' https: blob:; block-all-mixed-content; connect-src 'self' data: 'unsafe-inline' 'unsafe-eval' https: blob: https://*.trouter.io:443 https://*.trouter.skype.com:443 wss://*.trouter.io:443 wss://*.trouter.skype.com:443;
stackoverflow.com
upgrade-insecure-requests

Given that CSP is non trivial and policies can break the app, consider adding
report-uri "/csp-report-violation"
or use the free service report-uri
report-uri "https://report-uri.io/"
and view the results in developer tools and/or log the results
{"csp-report": {
    "document-uri": "https://example.com/signup.html",
    "referrer": "",
    "blocked-uri": "http://example.com/css/style.css",
    "violated-directive": "style-src cdn.example.com",
    "original-policy": "default-src 'none'; style-src cdn.example.com; report-uri /_/csp-reports",
    "disposition": "report"
}}

To test your changes, load you application or website in a browser and view the developer tools network tab and review the headers.
For nicer graphical view, with a warm fuzzy grade system, which others might use, visit
securityheaders.io


Examples of how to add the HTTP Security Headers

PHP:
header("X-Content-Type-Options: nosniff");
header("X-XSS-Protection: 1");
header("X-Frame-Options: sameorigin");
header("Strict-Transport-Security: max-age=31536000s; includeSubDomains");
header("Referrer-Policy: no-referrer-when-downgrade");
header("Access-Control-Allow-Origin: *");
header("Content-Security-Policy: base-uri 'self'; object-src 'none'");

PHP framework/library:
$response->headers->set('X-Content-Type-Options', 'nosniff');
$response->headers->set('X-XSS-Protection', '1');
$response->headers->set('X-Frame-Options', 'sameorigin');
$response->headers->set('Strict-Transport-Security', 'max-age=31536000s; includeSubDomains');
$response->headers->set('Referrer-Policy', 'no-referrer-when-downgrade');
$response->headers->set('Access-Control-Allow-Origin', '*');
$response->headers->set('Content-Security-Policy', "base-uri 'self'; object-src 'none'");

Apache config/.htaccess:
Header set X-Content-Type-Options nosniff
Header set X-XSS-Protection 1
Header set X-Frame-Options sameorigin
Header set Strict-Transport-Security max-age=31536000s; includeSubDomains
Header set Referrer-Policy no-referrer-when-downgrade
Header set Access-Control-Allow-Origin *
Header set Content-Security-Policy "base-uri 'self'; object-src 'none'"

nginx config
add_header X-Content-Type-Options nosniff
add_header X-XSS-Protection 1
add_header X-Frame-Options sameorigin
add_header Strict-Transport-Security max-age=31536000s; includeSubDomains
add_header Referrer-Policy no-referrer-when-downgrade
add_header Access-Control-Allow-Origin *
add_header Content-Security-Policy "base-uri 'self'; object-src 'none'"

-End of Document-

Thanks for reading

   

Mr. Roboto er Robocopy

Robocopy, or "Robust File Copy", is a command-line directory and/or file replication command
From the source directory, find all shortcuts, and copy the shortcut contents to a destination directory [Wikipedia]

The most basic example
Copies files from Directory_A to Directory_B
> robocopy C:\Directory_A C:\Directory_B

Some info about Robocopy defaults:
It will only copy a file if the source and destination have different time stamps or different file sizes.
Also, data, attributes, and time stamps are copied. ACL permissions, owner information,
and auditing information are ignored. You can modify this behavior with the /copy flag.
Also note that the paths don't have a trailing backslash.

The following flurry of options will mirror files form a source directory to a destination directory, recursively, with status, summary and a log

> robocopy /b /e /xa:s /xjd /sl /a-:hs /mt /fp /mir /mt:2 /log:"C:/dev/transfer.log" /eta /tee /v /l "C:/dev/source" "D:/backup/dev/source"     

And a translation of the options:
/b      - backup mode (there's a /zb option for restart mode, but it's a whole lot slower); overwrite acls
/e      - copies subdirectories (including empty directories) in addition to files
/xa:s   - exclude system files
/xjd    - exclude junction points
/sl     - copy symbolic links as links
/a-:hs  - remove hidden/system attributes from files
/fp     - full path of files in output
/mir    - MIRror a directory tree (equivalent to /e plus /purge)
/mt[:n] - Do multi-threaded copies with n threads (default 8)
/log:transfer.log - redirect output to file
/eta    - time remaining
/tee    - duplicate log to console window
/v      - verbose output + skipped
/l      - list files only (and not copy, delete, or time stamp)

Remove the option /l when ready to run

Note that the option /sl allows symbolic links to be copied, which is useful if you are using npm for node modules management or your git repository has symbolic links.

The options /eta, /tee, /v can be removed to minimize output to the console

You can also place the command within a bat file to run on double click, or some other event.

backup.bat:

echo "R: ramdrive -> C: backup"

robocopy /b /e /xa:s /xjd /sl /a-:hs /mt /fp /mir /mt:2 /log:"C:/backup/robocopy/transfer.log" /eta /tee /v /l "R:/code" "C:/backup/code" 

pause

Microsoft documentation of Robocopy and other possible arguments

-End of document-
Thanks for reading

PowerShell script to copy files based on shortcuts

From the source directory, find all shortcuts, and copy the shortcut contents to a destination directory

PowerShell is a task automation and configuration management framework from Microsoft, consisting of a command-line shell and associated scripting language. Wikipedia

# Get all shortcuts
$shortcuts = gci "$srcDir\*.lnk"

cgi = Get-ChildItem  Microsoft
Gets the items and child items in one or more specified locations.

# skip existing dirs
if (Test-Path "$destPath") {

Test-Path Microsoft
Determines whether all elements of a path exist.

# copy
copy-item -Path "$srcPath" -Destination "$destRecreatePath" -Force -Recurse -Container -Exclude $exclude

copy-item Microsoft
Copies an item from one location to another.
But alas, without any indication of progress.

So, from some help on Stack Overflow
# xcopy prompts for is this a file/dir, no progress
# robocopy asks for admin perms on ntfs/audit attribs
# copy copies with progress %
# /z   : Copies networked files in restartable mode.
cmd /c copy /z $srcFile $destFile

Some screenshots of the full script in action

And the full source is on GitHub

End of document. Thanks for reading.