How to get a struct back from an interface (returned by a factory) without type assertion?

I'm quite new to Go so some of it's concepts are still confusing. I'm trying to implement a Factory design pattern based on the reflect package so I'll be able to instantiate custom types dynamically.

I have a Factory that should return structs based on the content of a map. The map has as values instances implementing the reflect.Type interface. As I don't know what type I'm returning I have to return an interface implemented by all my types. The thing is I can't populate an interface, so I need to get the struct back.

Where I got so far (see it on the Go playground):

package main

import (
    "fmt"
    "reflect"
)

// Use a map to store types (typed nil values)
var Types = map[string]reflect.Type{
    "JSONDataSource": reflect.TypeOf((*JSONDataSource)(nil)),
    "XMLDataSource":  reflect.TypeOf((*XMLDataSource)(nil)),
}

func Factory(s string) (interface{}, error) {
    t := Types[s]
    // Can't use a type assertion here as I don't know the type yet.
    // I also tried reflect.New().Elem().Interface() but it doesn't change my problem
    return reflect.Zero(t).Interface(), nil
}

type Data interface {
    Import(config interface{}) error
    Export(config interface{}) error
}

type DataSource struct {
    filePath  string
    structPtr *interface{}
}

type JSONDataSource struct {
    *DataSource
}

func (d *JSONDataSource) Import(config interface{}) error {
    return nil
}

func (d *JSONDataSource) Export(config interface{}) error {
    return nil
}

type XMLDataSource struct {
    *DataSource
}

func (d *XMLDataSource) Import(config interface{}) error {
    return nil
}

func (d *XMLDataSource) Export(config interface{}) error {
    return nil
}

func main() {

    j, _ := Factory("JSONDataSource")
    // the following throws error since 'j' is an interface{}, not a 'JSONDataSource'
    j.filePath = "/my/path/"
    fmt.Println(reflect.TypeOf(j))

}

I've read a lot of topics, questions and articles during the past 24h but none of them helped me. Thank you by advance.

PS: I know I could do it simply with a switch case or dedicated NewXXX()functions for each type but that's not what I'm trying to achieve.