這邊紀錄閱讀 Hands on Machine Learning with Scikit-Learn & TensorFlow 的筆記。

Ch 1: The Machine Learning Landscape

機器學習是一種讓電腦從資料中學習的方法。

問題過於複雜（long lists of rules）且需要不斷找到新的pattern，或者是難以直接用傳統程式語言描述並解出的問題，就適合使用機器學習從資料中找出解法。
且透過機器學習的方法，人類也可以去檢視學習出來的模型，來得到該問題的相關知識。

根據機器學習的種類，可以分成：

1. Supervised / Unsupervised Learning ： 資料是否有標記解答
2. Batch and Online Learning ： Online為每筆資料進來，就會調整模型
3. Instance-Based Versus Model-Based Learning

這三大類可以交互組合。

Install Pyenv and Python 3

curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash

vim .bash_profile

add following lines to .bash_profile:

1
2
3

export PATH="/home/ec2-user/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

sudo yum install gcc

sudo yum install openssl-devel

sudo yum install bzip2-devel

sudo yum install sqlite-devel

sudo yum install readline-devel

pyenv install 3.4.3

Install lxml lib

sudo yum install libxml2-dev libxslt-dev python34-dev

初始設定

Hexo的官方網站，根據官方網站可以有基本的設定，但官方的介面不太好看，所以接下來要使用別人寫好的Theme。

when upgrade to xcode8, you may encounter some error in react native.

This link‘s method can fix in iOS 10. However, will be crach in iOS 8.4.

Finally, I use the method in this link.

(https://yanqiw.github.io/2016/10/05/react-native-adopt-to-xcode8.html)

http://stackoverflow.com/questions/39567555/new-build-disappears-after-uploading-it-to-itunes-connect

It will also email to your iTunes account.

fabric
ansible

nginx
gunicorn
systemd

celery
django-rq
rabbitmq/redis

General EB Settings

Use t2.small, otherwise the memory will not enough.

1
2
3
4
5
6
7
8
9

commands:
  000_dd:
    command: test -f /swapfile || dd if=/dev/zero of=/swapfile bs=1024 count=1500000
  001_mkswap:
    command: test -f /swapfile || mkswap /swapfile
  002_chmod:
    command: test -f /swapfile || chmod 0600 /swapfile
  003_swapon:
    command: test -f /swapfile || swapon /swapfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

packages:
  yum:
    git: []
    postgresql93-devel: []
    atlas-devel: []
    atlas-sse3-devel: []
    blas-devel: []
    gcc: []
    gcc-c++: []
    lapack-devel: []
    python34-devel: []
    libpng-devel: []
    freetype-devel: []
    geos: []
sources:
  /home/ec2-user: http://download.osgeo.org/gdal/1.11.2/gdal1112.zip
commands:
  04_install_gdal:
    cwd: /home/ec2-user/gdal-1.11.2
    command: test -e gdal-check.txt || ./configure --with-python=yes && make && sudo make install && echo 'ok' > gdal-check.txt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

container_commands:
  01_collectstatic:
    command: "source /opt/python/run/venv/bin/activate && source /opt/python/current/env && python manage.py collectstatic --noinput"
  02_migrate:
    command: "source /opt/python/run/venv/bin/activate && source /opt/python/current/env && python manage.py migrate --noinput"
    leader_only: true
  03_wsgi_custom:
    command: 'cp .ebextensions/wsgi.conf /opt/python/ondeck/wsgi.conf'
option_settings:
  "aws:elasticbeanstalk:application:environment":
    DJANGO_SETTINGS_MODULE: "xxx.settings_production"
    "PYTHONPATH": "/opt/python/current/app/xxx:$PYTHONPATH"
  "aws:elasticbeanstalk:container:python":
    WSGIPath: "xxx/wsgi.py"
  "aws:elasticbeanstalk:container:python:staticfiles":
    "/static/": "www/static/"

wsgi

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

LoadModule wsgi_module modules/mod_wsgi.so
WSGIPythonHome /opt/python/run/baselinenv
WSGISocketPrefix run/wsgi
WSGIRestrictEmbedded On
<VirtualHost *:80>
Alias /static/ /opt/python/current/app/www/static/
<Directory /opt/python/current/app/www/static/>
Order allow,deny
Allow from all
</Directory>
WSGIScriptAlias / /opt/python/current/app/xxx/wsgi.py
<Directory /opt/python/current/app/>
  Require all granted
</Directory>
WSGIDaemonProcess wsgi processes=1 threads=15 display-name=%{GROUP} \
  python-path=/opt/python/current/app:/opt/python/run/venv/lib64/python3.4/site-packages:/opt/python/run/venv/lib/python3.4/site-packages user=wsgi group=wsgi \
  home=/opt/python/current/app lang=en_US.UTF-8 locale=en_US.UTF-8
WSGIProcessGroup wsgi
</VirtualHost>
LogFormat "%h (%{X-Forwarded-For}i) %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
WSGIApplicationGroup %{GLOBAL}

Make sure place packages in the requirements.txt in the correct order (dependency). Place Scipy before scikit-learn.

Geopandas

requirements.txt Shapely==1.5.17.post1 => Shapely==1.5.17

eb deploy --timout 3600. Otherwise it will timeout to build gdal.

In AWS EB Web interface => configuration –> software configuration and edited the Environment Properties by adding the property LD_LIBRARY_PATH set as /usr/local/lib/:$LD_LIBRARY_PATH.

UTF-8

And if encounter the utf-8 encoding problems in EB.

1. You can add .encode('utf-8') if there are non-ascii char in the string.
2. Change wsgi.conf which in the /etc/httpd/conf.d/wsgi.conf

WSGIDaemonProcess wsgi processes=1 threads=15 display-name=%{GROUP} \
python-path=/opt/python/current/app:/opt/python/run/venv/lib64/python3.4/site-packages:/opt/python/run/venv/lib/python3.4/site-packages user=wsgi group=wsgi \
home=/opt/python/current/app lang=en_US.UTF-8 locale=en_US.UTF-8
WSGIProcessGroup wsgi

1. export LC_ALL=en_US.UTF-8 && export LANG=en_US.UTF-8 && command by using subprocess.call([“”], shell=True)

Reference:

http://stackoverflow.com/questions/25477268/script-timed-out-before-returning-headers-wsgi-py-on-elastic-beanstalk
https://code.google.com/archive/p/modwsgi/wikis/ApplicationIssues.wiki#Python_Simplified_GIL_State_API
http://stackoverflow.com/questions/30634777/unicodeencodeerror-when-running-in-mod-wsgi-even-after-setting-lang-and-lc-all
http://stackoverflow.com/questions/17326294/configuring-amazon-elastic-beanstalk-with-postgis
http://stackoverflow.com/questions/21715423/installing-postgis-on-amazon-elastic-beanstalk-using-config-file

製作
https://jackklpan.github.io/music_rank_player/

主要使用到以下東西
Lambda, Api Gateway, DynamoDB
vue.js
apex, (terraform)

Lambda串接googleapis和DynamoDB，查找歌曲在Youtube上面的播放Id，並存在DynamoDB，節省之後Youtube data api的查詢使用量。

Api Gateway串接Lambda，使前端可以用ajax query。

Apex的雷：

googleapis裡面有資料夾相對路徑的結構，因此不能和apex github範例上一樣，用webpack全部合併成一個檔案，相對路徑會錯誤。
我直接在functions裡面的資料夾（和index.js同一層的地方），npm install googleapis，同一層會連同一起zip上去。

Api Gateway的雷：

用terraform我覺得不好設定，直接用aws上面的介面比較容易。

在Method Request加上Query String

Integration Request加上Mapping，如下面網址說明，
使用application/json其中的預設template
http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html
但要修改escapeJavaScript，在兩個escapeJavaScript後面加上.replaceAll(“\‘“, “‘“)，如下面網址說明。否則當其中有 ‘ 符號的時候，會發生錯誤。
https://forums.aws.amazon.com/thread.jspa?threadID=228917

要加上CORS，使其他地方的domain也可以取用，如以下網址說明
http://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-cors.html

Youtube Api的雷：

其他的雷：

在Edge瀏覽器，要把script寫在一起才會Work…

First, install gettext

brew install gettext
brew link gettext --force

LOCALE_PATHS = (
    os.path.join(PROJECT_PATH, 'locale/'),
)

In your template,

1
2

{% load i18n %}
{% trans "XXX" %}

In command line,
django-admin makemessages -l zh-hant
django-admin.py compilemessages -l zh-hant

In urls.py

from django.conf.urls import include, url
from django.conf.urls.i18n import i18n_patterns
from django.contrib import admin

urlpatterns = [
    url(r'^admin/', include(admin.site.urls)),
]

urlpatterns += i18n_patterns(
    url(r'^$', 'home.views.index'),
)

Reference:
http://missions5.blogspot.tw/2015/03/django-1.html
http://blog.blackwhite.tw/2013/04/django-i18n.html
http://stackoverflow.com/questions/10280881/django-site-with-2-languages