API query for stock quote

stripped down to select columns: https://query.yahooapis.com/v1/public/yql?q=select Bid,Symbol from yahoo.finance.quotes where symbol in ("GLD,VTI")&format=json&env=store://datatables.org/alltableswithkeys&callback=

Other columns are Ask, Volume, or * to get all columns.

Sample ways to query with JAX-RS and GSON: http://www.skbh.in/restful-webservices/consuming-restful-web-services-with-the-jersey-client-api/

AWS EC2, RoR, ruby...

Created an Ubuntu EC2 instance, then followed https://gorails.com/setup/ubuntu/14.04 to install and setup ruby on rails.
Also, https://www.digitalocean.com/community/tutorials/how-to-setup-ruby-on-rails-with-postgres
for the postgres setup:
in config/database.yml set:
production:
  <<: *default
  database: myrailsapp_production
  username: <linux user account here, also create user with same user name in postgres db for "create role" command>
  password: <db password from "create role" command>

Added a security group "custom tcp rule" to permit port 3000 inbound. Wasn't getting a response from ruby on rails after running 'rails server'. netstat -tulpn showed it was listening on 127.0.0.1. To make listen on all ports followed https://fullstacknotes.com/make-rails-4-2-listen-to-all-interface/ , i.e. do instead: rails server -b 0.0.0.0

should try this later.

Hadoop, Avro experimentation

Followed http://hadooptutorial.info/avro-mapreduce-word-count-example/  but caution: I need to set the javac CLASSPATH instead to
export CLASSPATH="$HADOOP_HOME/share/hadoop/tools/lib/*"
export CLASSPATH="$HADOOP_HOME/share/hadoop/mapreduce/*:$CLASSPATH"

when I used the jar command it failed at first. This post helped to get it into alternatives: http://johnglotzer.blogspot.in/2012/09/alternatives-install-gets-stuck-failed.html

Used hadoop fs -put command to put the test.txt file onto the hadoop filesystem per http://hortonworks.com/hadoop-tutorial/using-commandline-manage-files-hdfs/

when running the MapReduceAvroWordCount application get errors similar to as described http://stackoverflow.com/questions/20586920/hadoop-connecting-to-resourcemanager-failed 

Added yarn.resourcemanager.address per answer.

jps command (see http://stackoverflow.com/questions/11738070/hadoop-cannot-use-jps-command) says hadoop services appear to be running.

stop/start per http://solaimurugan.blogspot.com/2014/05/step-by-step-instruction-how-startstop.html

this time didn't get same error, but job seems stuck.

Reading http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide/

issue stop-all.sh and modifying hadoop config files to match single-node installation.

tests under single-node Testing section gave same error as before: INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032.

Reinstalling to local user dir per alexjf blog.

Still attempts to connect to 0.0.0.0/0.0.0.0:8032

Reinstall Fedora, and set static ip address, Install JdK per http://tecadmin.net/install-java-8-on-centos-rhel-and-fedora/

enable ssh per https://docs.fedoraproject.org/en-US/Fedora/16/html/System_Administrators_Guide/s2-ssh-configuration-sshd.html

tried tests got error this time

Attempt to disable ipv6 using http://www.itsprite.com/linuxhow-to-disable-ipv6-networking-on-redhatcentosfedoraubuntu-linux-system/ but resulted in OS instability. Forced reboot.

could not login.

try reinstall OS.

install jdk, enable ssh. Tried enabling passwordless ssh per http://allthingshadoop.com/2010/04/20/hadoop-cluster-setup-ssh-key-authentication/

but would not work.

tried the hadoop installation and test from alexjf blog. It failed the test, pulling up the url, shows memory limit exceeded as cause of failure.

removed all settings from yarn-site.xml and stopped restarted using $HADOOP_PREFIX/sbin/stop-yarn.sh stop-dfs.sh start etc. but got seeming infinite running.

used https://coderwall.com/p/a5kbtw/installing-apache-hadoop-on-linux first set of instructions to enable passwordless ssh.

restart hadoop services and didn't ask for password this time. test still fails. 

modified command set --num_containers 1 --master_memory 512 and says completed successfully.

Trying on a Mac now per http://amodernstory.com/2014/09/23/installing-hadoop-on-mac-osx-yosemite/ works.

Back on Fedora setup; retried MapReduceAvroWordCount and it worked. Next try writing schema based on Java class: https://gist.github.com/QwertyManiac/4724582. How to convert schema to json representation (avsc file?) and use that instead of java class?

Setting up Authentication

Had an opendj installed on a Debian64 machine, just needed to reset the password for cn=Directory Manager
Followed http://allidm.com/blog/2012/09/change-password-for-directory-manager-in-forgerock-opendj/ just make sure when you copy into vi you ensure to delete the trailing white space heh.
Then I could run /opt/opendj/bin/status and enter the credentials successfully.
Tested connecting with Ldap admin using cn=Directory Manager
Connected. To set a user password under ou=Users, right click set Password, chose plain text and enter the password. Right click user again and select "copy dn to clipboard." Open another instance of the LDAP Admin tool, new connection, enter the IP and paste the dn of the user, enter the password, and verify with test connection.

In Java project create test class to connect following http://www.codejava.net/coding/connecting-to-ldap-server-using-jndi-in-java
Test same using and verify connected using dn and "simple" for the java.naming.security.authentication.


Installed MySQL and created a database and user on the Debian Wheezy VM per https://www.rackspace.com/knowledge_center/article/installing-mysql-server-on-debian

Following http://people.cis.ksu.edu/~hankley/d764/tut07/Nigusse_Spring.pdf now to populate a user info object from session. Get right maven dependencies from http://www.mkyong.com/spring/maven-spring-jdbc-example/ and http://examples.javacodegeeks.com/enterprise-java/spring/jdbc/spring-jdbctemplate-example/
Modify my.cnf to set bind-address to static ip. then /etc/init.d/mysql restart