DarkPAN over *anything_you_want* using cpanminus

Yesterday, I talked about a quick hack to add the SSH protocol using the scp URI scheme feature to cpanminus. At the end of the article, I added a technical note to explain how I saw the things to handle more protocols.

I just tried to do it.

With this version, cpanm can now handle scp:// and sftp:// URI schemes.

For scp://, it first tries to use the scp command then curl (hoping that curl is linked with libssh2).

For sftp://, it first tries to use the sftp command then curl (still hoping that curl is linked with libssh2).

It seems to work well.

I added the ability to develop external URI scheme backends. Imagine you want to add a new URI scheme, say foo://, without modifying App::cpanminus package.

To achieve this goal, create a module in the namespace App::cpanminus::backend::foo that will return a hash ref with two keys: get and mirror.

For each key, you have to provide an anonymous function to handle the action, like this:

package App::cpanminus::backend::foo;

{
    get => sub
    {
        my($self, $uri) = @_;
        # retrieve the content of the resource pointed by $uri (foo://...)
        # then return it
        return $content;
    },
    mirror => sub
    {
        my($self, $uri, $path) = @_;
        # retrieve the content of the resource pointed by $uri (foo://...)
        # and copy it to the file $path
        return $wathever_you_want;
    },
};
__END__

That’s all…

UPDATE 2011/10/2: As Paweł Murias suggested in comments below, I changed the “API” to a more standard one:

package App::cpanminus::backend::foo;

sub get
{
    my($self, $uri) = @_;
    # retrieve the content of the resource pointed by $uri (foo://...)
    # then return it
    return $content;
}

sub mirror
{
    my($self, $uri, $path) = @_;
    # retrieve the content of the resource pointed by $uri (foo://...)
    # and copy it to the file $path
    return $wathever_you_want;
}

1;
__END__

The hash ref old one in not available anymore.

DarkPAN over SSH using cpanminus

You want to set up a private CPAN, aka a DarkPAN, to put your own private modules in and deploy them as if they came from the official CPAN using cpanminus and its fantastic command cpanm.

The problem is that cpanminus works only with files, HTTP/HTTPS and FTP URIs. It is fine for all public uses, but for private ones, it is boring doing a setup of an HTTPS server with some authentication scheme, while SSH is so simple to use…

cpanm can use curl as backend and curl can handle scp/sftp if linked with libssh2. It’s OK, but too often in binary distros, curl is not linked with libssh2… and cpanm filters URIs based on the /^(ftp|https?|file):/ regexp… :( Too bad…

So as a quick hack and to avoid to depend on a specially compiled curl version, we modified cpanminus to make it support the scp protocol using the scp command. Quick hack visible on github.

On a server, create a darkpan user with a scponly shell. Using CPAN-Site set up a private CPAN as explained in its excellent documentation, at the root of the account.

Don’t forget to add the SSH public keys of all future clients of your DarkPAN into ~darkpan/.ssh/authorized_keys to avoid any password prompt.

Once your archive modules copied to ~darkpan/authors/id/M/MY/MYID and indexed via cpansite --site ~darkpan index, you can use the patched cpanminus module to access them:

cpanm -i --mirror scp://darkpan@example.com:. PrivateModule

Your private modules won’t be find on the official CPAN index, then cpanm will fall back to your mirror…

Easy, no?

If you want to test it, check out the cpanminus distribution then do a perl Makefile.PL --author, it will create the fat cpanm executable.

Technical note: to do a good work, it would be better to make cpanminus dispatches its mirror and get actions based on the protocol of the passed URI rather than on the availability of LWP, wget, curl or another module. One would avoid to patch each of these backends to detect file:// or scp://.

Proxy dispatcher for HTTP/SSL *and* SSH

Peteris Krumins just told us how he helps one of his friends to bypass a firewall to do SSH through the port 443 (HTTP/SSL one).

Last year, I did a proof of concept of a proxy that will listen on port 443 and forward the data on the internal HTTP server or SSH server, based on the client behavior without decoding anything.

To achieve this, I used AnyEvent, fantastic event loop manager…

One things to know is that when doing HTTP or HTTP over SSL, it is the client that first talk to the server, doing like:

GET /index.html HTTP/1.1
Host: www.ijenko.com
...

With SSH, the server announces itself, like that:

SSH-2.0-OpenSSH_5.4p1 FreeBSD-20100308

waiting then for client data…

So our proxy just has to wait a little bit after accepting the client connection (here we wait 0.5 seconds) before deciding what to do.

If the client talk during this time, it probably wants to do HTTP, if not it probably wants to do SSH.

The delay only impact SSH connections and only at the first step.

So reconfigure your HTTP server to only listen on localhost, then launch the proxy with the network side address.

Note that you can change the proxy to connect to different hosts than the local one (here 127.1), it’s up to you.

Enjoy… :-)

Just keep in mind that all connections to your internal HTTP and SSH servers will be coming from the proxy, you will not be able to know the real source, only the proxy knows…

use strict;
use warnings;

use AnyEvent;
use AnyEvent::Socket;
use AnyEvent::Handle;

die "usage: $0 BIND_IP_ADDRESS\n" if @ARGV != 1;

my $ip_address = shift;

use constant DEBUG => 1;

use constant {
    BIND_PORT   => 443,

    SSL_PORT    => 443,
    SSH_PORT    => 22,
};

tcp_server($ip_address, BIND_PORT, sub
           {
               my($fh, $host, $port) = @_;

               my $cnx = Cnx->new;

               $cnx->client_handle(
                   AnyEvent::Handle->new(
                       fh          => $fh,
                       rtimeout    => 0.5,
                       on_error    => $cnx->on_error,
                       # Client didn't say anything after initial timeout => SSH
                       on_rtimeout => $cnx->on_init_action(SSH_PORT),
                       # Client talk immediately => SSL
                       on_read     => $cnx->on_init_action(SSL_PORT)));

               warn "$host:$port connected.\n" if DEBUG;
           });


package Cnx;

use Scalar::Util qw(refaddr);

use AnyEvent;
use AnyEvent::Socket;
use AnyEvent::Handle;

use Carp;

my %CONNECTIONS;

sub new
{
    my($class, %opt) = @_;

    my $self = bless \%opt, $class;

    $CONNECTIONS{refaddr $self} = $self;

    return $self;
}


sub DESTROY
{
    my $self = shift;

    delete $CONNECTIONS{refaddr $self};

    warn "$self DESTROYed\n" if main::DEBUG;
}

# Create two accessors/mutators for attributes...
foreach my $attribute (qw(client_handle serv_handle))
{
    no strict 'refs';

    *$attribute = sub
    {
        if (@_ == 1)
        {
            return $_[0]{$attribute};
        }

        if (@_ == 2)
        {
            return $_[0]{$attribute} = $_[1];
        }

        carp "$attribute miscalled...";
    };
}


sub close_all
{
    my $self = shift;

    if (defined(my $handle = $self->client_handle))
    {
        $handle->destroy;
        $self->client_handle(undef);
    }

    if (defined(my $handle = $self->serv_handle))
    {
        $handle->destroy;
        $self->serv_handle(undef);
    }

    delete $CONNECTIONS{refaddr $self};
}


sub on_error
{
    my $self = shift;

    return sub
    {
        $self // return;

        my($handle, undef, $message) = @_;

        warn "CLIENT got error $message\n" if main::DEBUG;

        $self->close_all;
    };
}


sub on_init_action
{
    my($self, $port) = @_;

    # Something happens during the probe period
    return sub
    {
        my($handle, undef, $message) = @_;

        warn "$self on_init_action(PORT=$port).\n" if main::DEBUG;

        unless (defined $self->serv_handle)
        {
            # We cancel the timeout and we connect to the internal service
            $self->client_handle->rtimeout(0);

            tcp_connect('127.1', $port, $self->on_serv_connected($port));
        }
    };
}


sub on_client_read
{
    my $self = shift;

    # Client talk after the connection to the internal service
    return sub
    {
        my $handle = shift;

        warn "CLIENT -> serv: " . length($handle->{rbuf}) . " bytes\n"
            if main::DEBUG;

        $self->serv_handle->push_write(delete $handle->{rbuf});
    };
}


sub on_serv_connected
{
    my($self, $port) = @_;

    # We just connected to the internal service (or failed to)
    return sub
    {
        my $fh = shift;

        unless (defined $fh)
        {
            warn "Can't connect to internal service on port $port: $!\n";
            $self->close_all;
            return;
        }

        my $serv_handle = AnyEvent::Handle->new(
            fh => $fh,
            on_error => $self->on_serv_error,
            on_read  => $self->on_serv_read);

        warn "$serv_handle serv_connected\n" if main::DEBUG;

        $self->serv_handle($serv_handle);

        $self->client_handle->on_read($self->on_client_read);
    };
}


sub on_serv_error
{
    my $self = shift;

    # Error from internal service side
    return sub
    {
        my($serv_handle, undef, $msg) = @_;

        warn "SERV got error $msg\n" if main::DEBUG;

        $self->close_all;
    };
}


sub on_serv_read
{
    my $self = shift;

    # Something to read from internal service
    return sub
    {
        my $handle = shift;

        warn "SERV -> client: " . length($handle->{rbuf}) . " bytes\n"
            if main::DEBUG;

        $self->client_handle->push_write(delete $handle->{rbuf});
    };
}


package main;

AnyEvent->condvar->recv;