stripped down to select columns: https://query.yahooapis.com/v1/public/yql?q=select Bid,Symbol from yahoo.finance.quotes where symbol in ("GLD,VTI")&format=json&env=store://datatables.org/alltableswithkeys&callback=
Other columns are Ask, Volume, or * to get all columns.
Sample ways to query with JAX-RS and GSON: http://www.skbh.in/restful-webservices/consuming-restful-web-services-with-the-jersey-client-api/
Saturday, April 2, 2016
Tuesday, March 29, 2016
AWS EC2, RoR, ruby...
Created an Ubuntu EC2 instance, then followed https://gorails.com/setup/ubuntu/14.04 to install and setup ruby on rails.
Also, https://www.digitalocean.com/community/tutorials/how-to-setup-ruby-on-rails-with-postgres
for the postgres setup:
in config/database.yml set:
production:
<<: *default
database: myrailsapp_production
username: <linux user account here, also create user with same user name in postgres db for "create role" command>
password: <db password from "create role" command>
Added a security group "custom tcp rule" to permit port 3000 inbound. Wasn't getting a response from ruby on rails after running 'rails server'. netstat -tulpn showed it was listening on 127.0.0.1. To make listen on all ports followed https://fullstacknotes.com/make-rails-4-2-listen-to-all-interface/ , i.e. do instead: rails server -b 0.0.0.0
should try this later.
Also, https://www.digitalocean.com/community/tutorials/how-to-setup-ruby-on-rails-with-postgres
for the postgres setup:
in config/database.yml set:
production:
<<: *default
database: myrailsapp_production
username: <linux user account here, also create user with same user name in postgres db for "create role" command>
password: <db password from "create role" command>
should try this later.
Sunday, March 6, 2016
Hadoop, Avro experimentation
Followed http://hadooptutorial.info/avro-mapreduce-word-count-example/ but caution: I need to set the javac CLASSPATH instead to
export CLASSPATH="$HADOOP_HOME/share/hadoop/tools/lib/*"
export CLASSPATH="$HADOOP_HOME/share/hadoop/mapreduce/*:$CLASSPATH"
export CLASSPATH="$HADOOP_HOME/share/hadoop/tools/lib/*"
export CLASSPATH="$HADOOP_HOME/share/hadoop/mapreduce/*:$CLASSPATH"
when I used the jar command it failed at first. This post helped to get it into alternatives: http://johnglotzer.blogspot.in/2012/09/alternatives-install-gets-stuck-failed.html
Used hadoop fs -put command to put the test.txt file onto the hadoop filesystem per http://hortonworks.com/hadoop-tutorial/using-commandline-manage-files-hdfs/
when running the MapReduceAvroWordCount application get errors similar to as described http://stackoverflow.com/questions/20586920/hadoop-connecting-to-resourcemanager-failed
Added yarn.resourcemanager.address per answer.
jps command (see http://stackoverflow.com/questions/11738070/hadoop-cannot-use-jps-command) says hadoop services appear to be running.
this time didn't get same error, but job seems stuck.
issue stop-all.sh and modifying hadoop config files to match single-node installation.
tests under single-node Testing section gave same error as before: INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032.
Reinstalling to local user dir per alexjf blog.
Still attempts to connect to 0.0.0.0/0.0.0.0:8032
Reinstall Fedora, and set static ip address, Install JdK per http://tecadmin.net/install-java-8-on-centos-rhel-and-fedora/
enable ssh per https://docs.fedoraproject.org/en-US/Fedora/16/html/System_Administrators_Guide/s2-ssh-configuration-sshd.html
tried tests got error this time
Attempt to disable ipv6 using http://www.itsprite.com/linuxhow-to-disable-ipv6-networking-on-redhatcentosfedoraubuntu-linux-system/ but resulted in OS instability. Forced reboot.
could not login.
try reinstall OS.
install jdk, enable ssh. Tried enabling passwordless ssh per http://allthingshadoop.com/2010/04/20/hadoop-cluster-setup-ssh-key-authentication/
but would not work.
tried the hadoop installation and test from alexjf blog. It failed the test, pulling up the url, shows memory limit exceeded as cause of failure.
removed all settings from yarn-site.xml and stopped restarted using $HADOOP_PREFIX/sbin/stop-yarn.sh stop-dfs.sh start etc. but got seeming infinite running.
used https://coderwall.com/p/a5kbtw/installing-apache-hadoop-on-linux first set of instructions to enable passwordless ssh.
restart hadoop services and didn't ask for password this time. test still fails.
modified command set --num_containers 1 --master_memory 512 and says completed successfully.
Trying on a Mac now per http://amodernstory.com/2014/09/23/installing-hadoop-on-mac-osx-yosemite/ works.
Back on Fedora setup; retried MapReduceAvroWordCount and it worked. Next try writing schema based on Java class: https://gist.github.com/QwertyManiac/4724582. How to convert schema to json representation (avsc file?) and use that instead of java class?
Back on Fedora setup; retried MapReduceAvroWordCount and it worked. Next try writing schema based on Java class: https://gist.github.com/QwertyManiac/4724582. How to convert schema to json representation (avsc file?) and use that instead of java class?
Monday, January 18, 2016
Setting up Authentication
Had an opendj installed on a Debian64 machine, just needed to reset the password for cn=Directory Manager
Followed http://allidm.com/blog/2012/09/change-password-for-directory-manager-in-forgerock-opendj/ just make sure when you copy into vi you ensure to delete the trailing white space heh.
Then I could run /opt/opendj/bin/status and enter the credentials successfully.
Tested connecting with Ldap admin using cn=Directory Manager
Connected. To set a user password under ou=Users, right click set Password, chose plain text and enter the password. Right click user again and select "copy dn to clipboard." Open another instance of the LDAP Admin tool, new connection, enter the IP and paste the dn of the user, enter the password, and verify with test connection.
In Java project create test class to connect following http://www.codejava.net/coding/connecting-to-ldap-server-using-jndi-in-java
Test same using and verify connected using dn and "simple" for the java.naming.security.authentication.
Installed MySQL and created a database and user on the Debian Wheezy VM per https://www.rackspace.com/knowledge_center/article/installing-mysql-server-on-debian
Following http://people.cis.ksu.edu/~hankley/d764/tut07/Nigusse_Spring.pdf now to populate a user info object from session. Get right maven dependencies from http://www.mkyong.com/spring/maven-spring-jdbc-example/ and http://examples.javacodegeeks.com/enterprise-java/spring/jdbc/spring-jdbctemplate-example/
Modify my.cnf to set bind-address to static ip. then /etc/init.d/mysql restart
Followed http://allidm.com/blog/2012/09/change-password-for-directory-manager-in-forgerock-opendj/ just make sure when you copy into vi you ensure to delete the trailing white space heh.
Then I could run /opt/opendj/bin/status and enter the credentials successfully.
Tested connecting with Ldap admin using cn=Directory Manager
Connected. To set a user password under ou=Users, right click set Password, chose plain text and enter the password. Right click user again and select "copy dn to clipboard." Open another instance of the LDAP Admin tool, new connection, enter the IP and paste the dn of the user, enter the password, and verify with test connection.
In Java project create test class to connect following http://www.codejava.net/coding/connecting-to-ldap-server-using-jndi-in-java
Test same using and verify connected using dn and "simple" for the java.naming.security.authentication.
Installed MySQL and created a database and user on the Debian Wheezy VM per https://www.rackspace.com/knowledge_center/article/installing-mysql-server-on-debian
Following http://people.cis.ksu.edu/~hankley/d764/tut07/Nigusse_Spring.pdf now to populate a user info object from session. Get right maven dependencies from http://www.mkyong.com/spring/maven-spring-jdbc-example/ and http://examples.javacodegeeks.com/enterprise-java/spring/jdbc/spring-jdbctemplate-example/
Modify my.cnf to set bind-address to static ip. then /etc/init.d/mysql restart
Wednesday, December 30, 2015
SharePoint Emails
Send email per https://www.collaboris.com/blogs/collaboris-blog/mark-jones/2012/11/06/code-sample-how-to-programmatically-send-an-email-in-sharepoint#.VoH5A_krLak or using straight smtpclient per http://sharepoint.stackexchange.com/questions/98192/issue-when-setting-from-in-the-sputility-sendemail. In the end used this: http://www.sharepointbriefing.com/spcode/article.php/3840566/How-To-Send-Email-with-SPUtility.htm slightly modified:
private static Boolean SendEmail(SPWeb web, string email, string subject, string body)
{
try
{
bool flag = false;
SPSecurity.RunWithElevatedPrivileges(
delegate ()
{
flag = SPUtility.SendEmail(web, true, true,
email,
subject,
body);
});
return flag;
}
catch (System.Exception exp)
{
// Do some error logging
return false;
}
}
private static Boolean SendEmail(SPWeb web, string email, string subject, string body)
{
try
{
bool flag = false;
SPSecurity.RunWithElevatedPrivileges(
delegate ()
{
flag = SPUtility.SendEmail(web, true, true,
email,
subject,
body);
});
return flag;
}
catch (System.Exception exp)
{
// Do some error logging
return false;
}
}
SharePoint 2013 List Item Receiver Test and Deploy
Create a test list item event receiver per http://onceinawhilescribble.blogspot.com/2013/05/creating-simple-event-receiver-in.html
hit F5 or debug it to be sure it works as expected.
Deployment:
Right-click project node in Solution Explorer and choose publish.
Follow http://jaxsharepoint.blogspot.com/2013/06/deploying-wsp-to-sharepoint-2013.html and http://sharepoint.stackexchange.com/questions/140592/create-wsp-package-in-visual-studio-2013
to deploy the WSP to local server.
Verify sharepointproject1.wsp shows as deployed in http://myserver/_admin/Solutions.aspx
Get GUID of new feature, using this method http://iamprogrammerdotnet.blogspot.com/2012/03/enabledisable-feature-sharepoint-2010.html
or rename .wsp to .cab, extract the Feature.xml and copy the guid from the id attribute of the Feature element. Then enable the feature:
PS C:\Users\user1\Documents> enable-spfeature -identity "046347fa-aa0b-4a29-846e-eff84970a4fb" -url http://myserver/sites/teamsitecol1/
Verify it's activated in http://myserver/sites/teamsitecol1/_layouts/15/ManageFeatures.aspx and that it still works by creating new item in DocumentLog when adding item to target list.
In Visual Studio make small change to output that gets added to new list item. Do a "publish" as before. Then update the solution:
PS C:\Users\user1\Documents> update-spsolution -identity "sharepointproject1.wsp" -gacdeployment -literalpath "C:\Users\user1\Documents\sharepointproject1.wsp"
Repeat test and make sure new change is reflected.
Noticed I needed to rename the feature so that it uses a non-default name as shown in http://myserver/sites/teamsitecol1/_layouts/15/ManageFeatures.aspx
In Visual Studio, expand the feature and open the xml file; add an attribute called "Title" e.g. Title="My Feature" to the Feature element.
Then disable the feature in http://myserver/sites/teamsitecol1/_layouts/15/ManageFeatures.aspx and retract the existing solution from the SharePoint 2013 Management Shell:
uninstall-spsolution -identity "sharepointproject1.wsp"
remove-spsolution -identity "sharepointproject1.wsp" -force -confirm:$false
Then redploy as before.
hit F5 or debug it to be sure it works as expected.
Deployment:
Right-click project node in Solution Explorer and choose publish.
Follow http://jaxsharepoint.blogspot.com/2013/06/deploying-wsp-to-sharepoint-2013.html and http://sharepoint.stackexchange.com/questions/140592/create-wsp-package-in-visual-studio-2013
to deploy the WSP to local server.
PS C:\Users\user1\Documents> add-spsolution "C:\Users\user1\Documents\SharePointProject1.wsp" Name SolutionId Deployed ---- ---------- -------- sharepointproject1.wsp 51fbac1e-e23f-4dc4-a8c7-a9190577b206 False PS C:\Users\user1\Documents> install-spsolution -identity "sharepointproject1.wsp" -gacdeployment PS C:\Users\user1\Documents>
Verify sharepointproject1.wsp shows as deployed in http://myserver/_admin/Solutions.aspx
Get GUID of new feature, using this method http://iamprogrammerdotnet.blogspot.com/2012/03/enabledisable-feature-sharepoint-2010.html
or rename .wsp to .cab, extract the Feature.xml and copy the guid from the id attribute of the Feature element. Then enable the feature:
PS C:\Users\user1\Documents> enable-spfeature -identity "046347fa-aa0b-4a29-846e-eff84970a4fb" -url http://myserver/sites/teamsitecol1/
Verify it's activated in http://myserver/sites/teamsitecol1/_layouts/15/ManageFeatures.aspx and that it still works by creating new item in DocumentLog when adding item to target list.
In Visual Studio make small change to output that gets added to new list item. Do a "publish" as before. Then update the solution:
PS C:\Users\user1\Documents> update-spsolution -identity "sharepointproject1.wsp" -gacdeployment -literalpath "C:\Users\user1\Documents\sharepointproject1.wsp"
Repeat test and make sure new change is reflected.
Noticed I needed to rename the feature so that it uses a non-default name as shown in http://myserver/sites/teamsitecol1/_layouts/15/ManageFeatures.aspx
In Visual Studio, expand the feature and open the xml file; add an attribute called "Title" e.g. Title="My Feature" to the Feature element.
Then disable the feature in http://myserver/sites/teamsitecol1/_layouts/15/ManageFeatures.aspx and retract the existing solution from the SharePoint 2013 Management Shell:
uninstall-spsolution -identity "sharepointproject1.wsp"
remove-spsolution -identity "sharepointproject1.wsp" -force -confirm:$false
Then redploy as before.
Monday, December 28, 2015
SharePoint Outgoing Email to Exchange
Setting up SharePoint 2013 single-server farm to connect to Exchange 2016.
After installing prerequisites in a new Windows Server 2012 R2 (and installing all available windows updates after installation failed first time) per https://technet.microsoft.com/EN-US/library/bb125224(v=exchg.160).aspx run Setup.exe per https://technet.microsoft.com/EN-US/library/bb124778(v=exchg.160).aspx
Needed to run setup.exe a few times due to failures connecting to the domain controller, each time it would pick up where it left off.
After a final reboot, open Exchange Management Shell from all apps.
To get the autodiscover.xml do:
Get-ClientAccessService | fl AutoDiscoverServiceInternalUri
for OWA and ECP check the servers / virtual directories tab in Exchange Administrative Center from all apps AKA 'ecp'
In ECP you can also add mail boxes from the recipients tab.
Found out the frequent disconnects to the server could be fixed by changing the connection on the DC to wired from wireless.
Trying to figure out why outlook wouldn't connect. Tried https://myfqdnexchange/Autodiscover/Autodiscover.xml but got an error about "Could not load file or assembly ‘Microsoft.Exchange.Security'"
Taking a cue from https://exchangeranger.wordpress.com/2015/06/12/exchange-2013-cu8-application-event-warning-1310-microsoft-exchange-security-assembly/ copied SharedWebConfig.config from C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy to C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess
Then it gave me a 600 error which is actually indicative that everything is working! And sure enough, able to auto connect through Outlook (after deleting the corrupted profile from previous attempts from control panel, mail).
Note: I was able to connect from Outlook 2016. I added a host file entry with autodiscover.mydomain.com x.x.x.x but I'm not sure if that was necessary. Regardless I couldn't connect from Outlook 2013 (coudn't find server). But I was able to connect to owa via https://servername.mydomain.com/owa.
Then followed: http://sharepointgeorge.com/2010/configuring-outgoing-email-sharepoint-2010/
For the receiver connector selected "Frontend Transport" instead of Hub Transport because I heard on https://www.youtube.com/watch?v=73UBX_oJfgk that the latter is buggy.
For the CA Outgoing E-mail settings part, following https://alinimer.wordpress.com/2014/09/25/configure-outgoing-e-mail-for-sharepoint-20102013-to-using-office-365-or-gmail-relay/ instead, i.e. will relay through local SMTP server service (which is set in Services to start automatically), this is in case I wish to ditch Exchange and go with gmail in the future. For outbound security part, I will leave anonymous for now to connect to Exchange. So for Outbound Connections leave at port 25 (Exchange). For smart host, entered the FQDN of the Exchange server. For SMTP E-mail setting in IIS part, checked "Use localhost" box instead of entering the FQDN of the local server.
Set up an email address for a test user: Application Management, Manage Service Applications, User Profile Service Application, Manage User Profiles and add the test user's exchange email as the Work email property. Had to wait around 1/2 hour before it would show up as the test user's email under "Send me alerts by:" in the tasks list "New Alert" screen. Thought about enabling AD synchronization but it looked to complicated and outside of scope of this lab. Verified alert was sent to test user upon test user creating alert and task created. Couldn't do the "Send e-mail when ownership is assigned" test because apparently that feature was removed from SharePoint 2013, see http://sharepoint.stackexchange.com/questions/65056/send-email-when-ownership-is-assigned-due-date-functionality.
Followed https://technet.microsoft.com/en-us/library/jj657457(v=exchg.160).aspx to allow sending to non-exchange email addresses associated with user profiles in SharePoint.
After installing prerequisites in a new Windows Server 2012 R2 (and installing all available windows updates after installation failed first time) per https://technet.microsoft.com/EN-US/library/bb125224(v=exchg.160).aspx run Setup.exe per https://technet.microsoft.com/EN-US/library/bb124778(v=exchg.160).aspx
Needed to run setup.exe a few times due to failures connecting to the domain controller, each time it would pick up where it left off.
After a final reboot, open Exchange Management Shell from all apps.
To get the autodiscover.xml do:
Get-ClientAccessService | fl AutoDiscoverServiceInternalUri
for OWA and ECP check the servers / virtual directories tab in Exchange Administrative Center from all apps AKA 'ecp'
In ECP you can also add mail boxes from the recipients tab.
Found out the frequent disconnects to the server could be fixed by changing the connection on the DC to wired from wireless.
Trying to figure out why outlook wouldn't connect. Tried https://myfqdnexchange/Autodiscover/Autodiscover.xml but got an error about "Could not load file or assembly ‘Microsoft.Exchange.Security'"
Taking a cue from https://exchangeranger.wordpress.com/2015/06/12/exchange-2013-cu8-application-event-warning-1310-microsoft-exchange-security-assembly/ copied SharedWebConfig.config from C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy to C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess
Then it gave me a 600 error which is actually indicative that everything is working! And sure enough, able to auto connect through Outlook (after deleting the corrupted profile from previous attempts from control panel, mail).
Note: I was able to connect from Outlook 2016. I added a host file entry with autodiscover.mydomain.com x.x.x.x but I'm not sure if that was necessary. Regardless I couldn't connect from Outlook 2013 (coudn't find server). But I was able to connect to owa via https://servername.mydomain.com/owa.
Then followed: http://sharepointgeorge.com/2010/configuring-outgoing-email-sharepoint-2010/
For the receiver connector selected "Frontend Transport" instead of Hub Transport because I heard on https://www.youtube.com/watch?v=73UBX_oJfgk that the latter is buggy.
For the CA Outgoing E-mail settings part, following https://alinimer.wordpress.com/2014/09/25/configure-outgoing-e-mail-for-sharepoint-20102013-to-using-office-365-or-gmail-relay/ instead, i.e. will relay through local SMTP server service (which is set in Services to start automatically), this is in case I wish to ditch Exchange and go with gmail in the future. For outbound security part, I will leave anonymous for now to connect to Exchange. So for Outbound Connections leave at port 25 (Exchange). For smart host, entered the FQDN of the Exchange server. For SMTP E-mail setting in IIS part, checked "Use localhost" box instead of entering the FQDN of the local server.
Set up an email address for a test user: Application Management, Manage Service Applications, User Profile Service Application, Manage User Profiles and add the test user's exchange email as the Work email property. Had to wait around 1/2 hour before it would show up as the test user's email under "Send me alerts by:" in the tasks list "New Alert" screen. Thought about enabling AD synchronization but it looked to complicated and outside of scope of this lab. Verified alert was sent to test user upon test user creating alert and task created. Couldn't do the "Send e-mail when ownership is assigned" test because apparently that feature was removed from SharePoint 2013, see http://sharepoint.stackexchange.com/questions/65056/send-email-when-ownership-is-assigned-due-date-functionality.
Followed https://technet.microsoft.com/en-us/library/jj657457(v=exchg.160).aspx to allow sending to non-exchange email addresses associated with user profiles in SharePoint.
Subscribe to:
Posts (Atom)