13 September 2016

Rename tv series movies files and subtitles to match

I wrote a simple script to rename the tv series files and its subtitles and remove all extract characters but the episode number along with the extension so the media players (such as VLC) can assign the subtitles automatically.

https://gist.github.com/MuhammadHewedy/a23225acc389e25d393df4659123b880

05 September 2016

shell script to import certificates into java cacerts

I am not the original author, I just some small enhancements



 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#! /bin/bash

if [ $# -eq 0 ]; then
    echo -e "usage: $0 <host>\nexample: $0 abc.com"
    exit -1
fi

KEYTOOL=../../bin/keytool
HOST=$1
PORT=443
KEYSTOREFILE=cacerts
KEYSTOREFILE_BKUP=$KEYSTOREFILE.`date '+%Y%m%d%H%M'`.'original'
KEYSTOREPASS=changeit

if [ ! -f $KEYSTOREFILE ]; then
    echo -e "You must run this script from the directory jdk/jre/lib/security"
    exit -1
fi

#backup the cacerts file
echo -e "\n\n**** BAKCING UP THE $KEYSTOREFILE TO $KEYSTOREFILE_BKUP ****\n\n"
cp $KEYSTOREFILE $KEYSTOREFILE_BKUP


# get the SSL certificate
echo -e "\n\n**** SAVING THE CERTIFCATE TO ${HOST}.cert ****\n\n"
openssl s_client -connect ${HOST}:${PORT} </dev/null \
    | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ${HOST}.cert

echo -e "\n\n**** USING keytool AT $KEYTOOL ****\n\n"

# create a keystore and import certificate
echo -e "\n\n**** IMPORTING THE CERTIFICATE... ****\n\n"
"$KEYTOOL" -import -noprompt -trustcacerts \
    -alias ${HOST} -file ${HOST}.cert \
    -keystore ${KEYSTOREFILE} -storepass ${KEYSTOREPASS}

echo -e "\n\n**** PRINTING THE CERTIFICATE AFTER IMPORTED ... ****\n\n"
# verify we've got it.
"$KEYTOOL" -list -v -keystore ${KEYSTOREFILE} -storepass ${KEYSTOREPASS} -alias ${HOST} | grep --color=always $HOST

Using @Transactional to trigger rollback on runtime exceptions

In the last post, I've discussed some thoughts about trying to understand spring transactions.

Although not complete (and it appears it has issues as well). but today's topic is so compact and clear.

Spring follow's EJB transaction on that, it rollback on RuntimeException.

Suppose we have the following method:

@Transactionl
public void myServiceMethod(){
    updateDatabase();
    callSomeQueueOrWebService();
}

so, if the the second method call fails with RuntimeException, then the whole transaction will rollback causing the first method to rolled back.

But if you don't annotate the method with @Transactional, then both method calls (updateDatabase() and callSomeQueueOrWebService() ) will run independently and one might success however the other case fail.

Code:

Service Layer:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
private String fName = "Mohammad", lName = "Hewedy";

 @Transactional
 public void updateDbAndCallWSTrans() {
  userRepo.save(User.of(null, fName, lName));
  callWsThatThrowsException();
 }

 public void updateDbAndCallWSNoTrans() {
  userRepo.save(User.of(null, fName, lName));
  callWsThatThrowsException();
 }

 public Stream<User> getDbUpdate() {
  return userRepo.findByFNameAndLName(fName, lName);
 }

 private void callWsThatThrowsException() {
  // automatic rollback for runtime exceptions only!
  throw new RuntimeException("EXCEPTION from WS");
 }


The controller layer:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
 
 @RequestMapping("/tx/user")
 public ResponseEntity<?> getDbUpdate() {
  return ResponseEntity.ok(userService.getDbUpdate().toArray(User[]::new));
 }

 @RequestMapping("/tx/add-w")
 public ResponseEntity<?> updateDbAndCallWSTrans() {
  userService.updateDbAndCallWSTrans();
  return ResponseEntity.ok().build();
 }

 @RequestMapping("/tx/add-wo")
 public ResponseEntity<?> updateDbAndCallWSNoTrans() {
  userService.updateDbAndCallWSNoTrans();
  return ResponseEntity.ok().build();
 }


Note, the call to /tx/add-w will not insert any data into the database, and calls to /tx/user will return empty.

However, the call to /tx/add-wo, will insert the user object into the database, and call to /tx/user will return the inserted users object.

04 September 2016

My thoughts about Spring Transactions

I've some thoughts about spring (and spring-boot) transactions I want to list somewhere, and I think here's a nice place.

Preface

Spring do transaction management through PlatformTransactionManager interface.

This interface has implementation for different kinds of datasources, for example for DataSourceTransactionManagerHibernateTransactionManagerHibernateTransactionManagerJmsTransactionManagerJpaTransactionManager and others.

In a typical JPA application, the JpaTransactionManager is being used.

In SpringBoot, All the JpaRepository are annotated with @Transactional annotation.

This annotation used by spring to do the Declarative transaction management.

So, the Developer uses this annotation to control many things, included the transaction propagation (REQUIRED, REQUIRED_NEW, SUPPORTED, etc), the isolation level (READ_COMMITED, READ_UNCOMMITED, etc.) and the rollback exception, timeout and more.

And behind the since, the platform-specific transaction manager along with the databasource do the actual work.

@Transactionl

So, many people says, use the @Transactional annotation on the service layer, you might need to use it on DAO (now called Repository) layer, but do not use it on the controller layer.

In spring boot, as I said, all the out-of-the-box methods that you get by implementing the JpaRepository are already have the @Transactional annotation (see SimpleJpaRepository). And I think hence the @Transactional on the implementation class itself, then all your repo methods might have the annotation as well.

But this is not the problem, the problem in the Service layer.

@Transactionl in the Service layer

What if I didn't put the Transactional annotation on my service methods?
Then your method will not run in a transaction, so If you have a service method that do the following

public void serviceMethod(){
     insert1();
     insert2();
}

insert1() might succeed and inserts data into the database, while insert2() fails.

In this case you should put the @Transactional annotation on your service method if you want to have all or none.

@Transactionl and noRollbackFor

The rollback is controlled by RuleBasedTransactionAttribute
see http://docs.spring.io/autorepo/docs/spring/current/spring-framework-reference/html/transaction.html#transaction-declarative-rolling-back

In its default configuration, the Spring Framework’s transaction infrastructure code only marks a transaction for rollback in the case of runtime, unchecked exceptions; that is, when the thrown exception is an instance or subclass of RuntimeException. ( Errors will also - by default - result in a rollback). Checked exceptions that are thrown from a transactional method do not result in rollback in the default configuration.
Also,
When the Spring Framework’s transaction infrastructure catches an exception and is consults configured rollback rules to determine whether to mark the transaction for rollback, the strongest matching rule wins 

For rollbackFor and noRollbackFor;

I think spring take the exception thrown and check to see how it is close (in inheritance hierarchy) to the declared exceptions (in rollbackFor and noRollbackFor). If it is closer to an exception declared in rollbackFor more than one declared in noRollbackFor, then the transaction will be rollbacked. and vice versa.

ex:
We have this heirarcy:
class Exception1{}
class Exception2 extends 1{}
class Exception3 extends 2{}

@Transactional(rollbackFor=Exception1.class, noRollbackFor=Exception2 .class)
public void myMethod(){
        throw new Exception3 ();
}

In this case, the transaction will NOT be rollbacked, because the thrown exception (Exception3) is closest to exception in noRollbackFor (Exception2) than the exception in rollbackFor (Exception1)

And if the closest exception declared in rollbackFor, the spring will rollback the tx, otherwise it will not rollback it:

in RuleBasedTransactionAttribute: return !(winner instanceof NoRollbackRuleAttribute);

@Transactionl and lazy evaluation

lazy evaluation is important aspect in web development, it facilities things, lazy evaluation includes not only evaluate lazy associations in the Controller layer, but also evaluation lazy Streams. (Thanks to OpenEntityManagerInViewFilter).

But if you choose to annotate your service method with @Transactional, lazy transaction data will not be available in the Controller layer (and exceptions will be thrown as well).

Here's an example issue.

So, some people says, at service layer get all the data you want and then send it to the controller layer as eager so you don't need the service layer's transaction anymore.

I think for finder services, you might not need to have the @Transactional annotation, so you can make use of the OpenEntityManagerInViewFilter (which is provided by springboot by default) and be able to  enjoy your laziness in the controller layer.(It seems the "Open EntityManager in View" has nothing to do here, it only allow the transaction manager to use an EntityManager created by him. because if you have @Transactional and have this pattern on, the exception will continue to throw if you try to access lazy data out of the transactions.

But in case of services that updates the database, you do need to use the @Transactional annotation to control the things not to go messy!

@Transactionl on controller and service layer

If you have a finder method that returns a lazy object (java 8 stream or lazy association as in hibernate), the if you have the @Transactional annotation on the service layer, then the lazy parts of  object will not be available to the controller layer.

However, if you put the @Transactionl annotation on the controller layer as well (which many people don't recommend) then the lazy part will be available to the controller layer unless you make the transaction propagation on the service layer as REQUIRED_NEW, in this case the service layer will suspend the controller's transaction and start/end it's own transaction and will send back the object to the controller out of its transaction and the problem will still exists.

Conclusion

For now, I'll  - mostly, because some times I do not need it - use @Transactionl on Service methods that do update (Add/update/delete) the database.

However for finder service methods (that query the database), it is better to have the @Transational(readOnly=true) on it, but since I didn't understand - till now- much more, I'll choose not to use this annotation so I can deal with my laziness objects in the controller layer)

Interesting classes to see the source code of:
1. TransactionInterceptor: class source code to see what is happening internally. (very interesting)
2. TransactionAspectSupport:
  a. method: invokeWithinTransaction
  b. method: completeTransactionAfterThrowing (it shows
3. AbstractPlateformTransactoinManager
 a. method: commit (it shows the rollback if done if "setRollbackOnly" method is being called

The post above is based on my thoughts and is might not be valid.

31 August 2016

callback on thread completion

The code illustrate how callback is being accomplished on Thread completion:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
class Executor {

  public <T> void execute(Callable<T> task, BiConsumer<T, ? super Exception> callback) {
     CallableThread<T> thread = new CallableThread<>(task, callback);
     thread.start();
  }

  class CallableThread<T> extends Thread {

     Callable<T> task;
     BiConsumer<T, ? super Exception> callback;

     public CallableThread(Callable<T> task, BiConsumer<T, ? super Exception> callback) {
        this.task = task;
        this.callback = callback;
     }

     public void run() {
        System.out.println("running task on thread : " + Thread.currentThread().getName());
        try {
            T t = task.call();
            callback.accept(t, null);
        } catch (Exception ex) {
            callback.accept(null, ex);
        }
     }
   }
 }

Caller:
1
2
3
4
5
6
7
8
System.out.println("running task on thread : " + Thread.currentThread().getName());

new Executor().execute(() -> "HELLO WORKD", (result, ex) -> {
    System.out.println("result: " + result);
    System.out.println("exception: " + ex);
});

System.out.println("finished running task on thread : " + Thread.currentThread().getName());


Output:
1
2
3
4
5
running task on thread : main
finished running task on thread : main
running task on thread : Thread-0
result: HELLO WORKD
exception: null

It appears that, the Executor is submitting the Callable to execution and on complete it invokes the callback functional interface.


30 July 2016

Hide and Unhide Plex Media Servers Libraries

Plex media server is a famous media server, I use it at home to set local media server for myself and family.

However, to be able to hide some Libraries is not straightforward,  as it is not supported by the app itself, and there's a plugin that allows you to do so for 1 Library for free and pay for the others (It called Lock for Plex).

I used this extension and find out how it works, and I hacked it and be able to hide more than one Library.

How "Lock for Plex" work is by updating the database used by Plex.

It uses two bash files, one for lock and the other for unlock:


Plex Media Server/Plug-ins/Lock_for_Plex.bundle/Contents/Resources/lock_support/lock.sh

Plex Media Server/Plug-ins/Lock_for_Plex.bundle/Contents/Resources/lock_support/unlock.sh

Here's the content for myself after I've been modified on this file:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cat lock.sh
#!/bin/bash


lockplex="UPDATE metadata_items SET metadata_type=20 WHERE library_section_id=2 and metadata_type=1; DELETE FROM library_sections WHERE id=2;"
lockplex2="UPDATE metadata_items SET metadata_type=4444 WHERE library_section_id=8 and metadata_type=4; DELETE FROM library_sections WHERE id=8;"

cd "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Lock_for_Plex.bundle/Contents/Resources/lock_support/"

./sqlite3 "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db" "$lockplex" 
./sqlite3 "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db" "$lockplex2" 

exit;
mhewedy@probook:~$ cat unlock.sh 
#!/bin/bash

unlockplex="UPDATE metadata_items SET metadata_type=1 WHERE library_section_id=2 and metadata_type=20; INSERT OR REPLACE INTO library_sections (id,name,section_type,language,agent,scanner,created_at,updated_at,scanned_at,user_fields,uuid) VALUES (2,'English Movies',1,'en','com.plexapp.agents.imdb','Plex Movie Scanner','2016-07-03 07:27:07','2016-07-03 08:35:21','2016-07-03 08:35:21','pr%3AenableCinemaTrailers=0&pr%3AincludeInGlobal=0','c07640f7-a137-4987-95ba-c3c5850383d3');"

unlockplex2="UPDATE metadata_items SET metadata_type=4 WHERE library_section_id=8 and metadata_type=4444; INSERT OR REPLACE INTO library_sections (id,name,section_type,language,agent,scanner,created_at,updated_at,scanned_at,user_fields,uuid) VALUES (8,'TV Shows',2,'en','com.plexapp.agents.thetvdb','Plex Movie Scanner','2016-07-07 03:45:33','2016-07-07 03:45:55','2016-07-13 21:03:02','pr%3AincludeInGlobal=0','0d670ea9-0da4-4639-8f52-2fcd64f3d8c8');"

cd "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Lock_for_Plex.bundle/Contents/Resources/lock_support/"

./sqlite3 "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db" "$unlockplex" 
./sqlite3 "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db" "$unlockplex2" 

exit;

One other way to achieve the same result is to have some other disk partition and mount it on-demond.


26 July 2016

Create Local git server

Actually git doesn't work with client-server concept, it is distributed source control, where any repo can act as a remote repo for others and in the same time a local repo.

But in your organization, you need to have a place where you have your code resides either for keep code save, or even to do automated deployment or for any other reason.

suppose you have 2 Linux machines, machine A and machine B. (machine A will act like a server, where machine B will be your development machine)

1. machine A have to have ssh server installed; Login to machine A with user say 'git':

ssh git@machine_a

2. go to a directory where you want to keep your code, or even you want to do the autodeployemnt from there

sudo mkdir -p /repos/test-repo
sudo chown -R git:git /repos/rest-repo

3. init the origin on the server

cd /repos/test-repo
git init --bare

4. Your server is ready now, Now go to the client machine and create a folder where you will put your development source code:

cd ~/my-test-repo-code

5. create local git repo and point to the remote server we just created

git init
git remote add origin git@machine_a:/repos/test-repo

6. test your installation:

# we still in the client machine
touch test-file.txt
git add test-file.txt
git commit -a -m 'init'
git push remote origin

7. that's it, you now can write code and push it to your server, you can add hocks on the server (for deployment) or install your a web server to show the code in fancy interface or support other activities like code-review, etc.

Have fun.